Object observation system and method of controlling object observation system

ABSTRACT

An object observation system of the present invention has an observation apparatus for observing a body to be examined, a three-dimensional image recording apparatus for recording three-dimensional images, which are obtained in advance, of the body to be examined, and an image constructing apparatus for constructing a three-dimensional image based on images in synchronization with the observation apparatus, which are recorded in the three-dimensional image recording apparatus.

This application claims benefit of Japanese Application Nos. 2003-157041filed on Jun. 2, 2003, 2003-157042 filed on Jun. 2, 2003, 2003-189784filed on Jul. 1, 2003, 2003-189785 filed on Jul. 1, 2003, 2004-024828filed on Jan. 30, 2004, 2004-024829 filed on Jan. 30, 2004, 2004-024830filed on Jan. 30, 2004, 2004-024831 filed on Jan. 30, 2004, 2004-024832filed on Jan. 30, 2004, 2004-024833 filed on Jan. 30, 2004, the contentsof which are incorporated by this reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an object observation system and amethod of controlling an object observation system.

2. Description of Related Art

Endoscope apparatuses have been widely used in medical fields andindustrial fields. In the endoscope apparatus, an endoscopic imageobtained by a television camera externally installed endoscope in whicha television camera is attached to an eyepiece portion of an opticalendoscope or an electronic endoscope self-containing an image pickupapparatus at the distal end of an insert portion thereof is displayed ona monitor. With reference to the endoscopic image, observation and/ortreatment may be performed.

An endoscopic surgery system using the endoscope apparatus is used forperforming an operation under endoscopic observation by using apneumoperitoneum apparatus and/or a high-frequency cautery apparatus,for example, as multiple peripheral apparatuses in addition to a cameracontrol unit (called CCU or video processor, hereinafter) including alight source apparatus for supplying illumination light to an endoscopeand/or a video signal processing circuit for displaying endoscopicimages and/or a TV monitor for displaying endoscopic images. In theendoscopic surgery system, the multiple peripheral apparatuses areconnected to a system controller in order to centrally control themultiple peripheral apparatuses.

With the recent increase in processing speed of computers, theendoscopic surgery system can reconstruct a volume rendering image(simply called rendering image or VR image, hereinafter) as a virtualthree dimensional image (called virtual image, hereinafter) instantly byusing medical image data in a three-dimensional area and can display arendering image on a display screen of the monitor as a navigation imagefor guiding an endoscope, for example, to a target part of a body to beexamined and/or a reference image for checking the surroundings of atarget part.

As this kind of conventional endoscopic surgery system, a system used ina bronchial endoscope apparatus has been proposed as disclosed inJapanese Unexamined Patent Application Publication No. 2000-135215.

The endoscopic surgery system disclosed in the publication creates athree dimensional image of a tract within a body to be examined based onmedical image data in a three-dimensional area of the body to beexamined, obtains a path to a target point along the tract on thethree-dimensional image, creates a virtual rendering image of the tractalong the path based on the medical image data and displays the virtualrendering image on the monitor. Thus, a bronchial endoscope can beguided or navigated to a target part.

With the endoscopic surgery system used in the bronchial endoscopeapparatus, a rendering image of a predetermined path is displayed. Inthis case, an operator does not have to operate or instruct in themiddle in particular. Therefore, the endoscopic surgery system is usefulfor navigating the bronchial endoscope to a tract in the body such asthe bronchial tubes, which limits a direction of line of vision.

On the other hand, a conventional endoscopic surgery system can displaya rendering image as a reference image in addition to an endoscopicimage when the conventional endoscopic surgery system is used forsurgery.

Generally, in surgery, an operator performs surgical treatment by usinga treating device such as an electric knife with reference to anendoscopic image. In this case, the operator uses rendering images of atarget part and the surroundings as reference images in order to checkthe state of the blood vessels around an internal organ and/or the backof an internal organ.

Therefore, the endoscopic surgery system must display a rendering imageas a reference image that an operator needs to check on the spot duringsurgery more than a case where a rendering image is used for navigationof a bronchial endoscope, for example.

Therefore, the conventional endoscopic surgery system displays arendering image in response to a manipulation on a mouse and/or akeyboard by a nurse or an operator in an unclean area based on aninstruction by an operator in a clean area.

Recently, in a surgical operation, various progressions and results ofthe surgery are often recorded, and endoscopic images may be alsorecorded. During surgery, an operator takes photographs by manipulatinga release switch in order to store in patient's charts and records andsaves still-image data of endoscopic images as a record of the surgery.

In order to create a three-dimensional image as described above, athree-dimensional virtual image data of the inside of a body to beexamined is obtained by picking up a tomographic image of the body to beexamined by using an X-ray computed topography (CT) apparatus, forexample. Thus, an affected part can be diagnosed by using the virtualimage data.

In the CT apparatus, X-ray irradiation/detection are rotatedcontinuously, and, at the same time, a body to be examined is fed inseries in the body axis direction. Thus, continuous helical scanning canbe performed on a three-dimensional area of the body to be examined.Therefore, a three-dimensional virtual image can be created fromtomographic images of serial slices of the three-dimensional area.

One of this kind of three dimensional image is a three dimensional imageof the bronchi of the lung. A three-dimensional image of the bronchi isused for three-dimensionally identifying a position of an abnormal part,which is suspected as a lung cancer, for example. In order to check theabnormal part by performing a biopsy, a bronchial endoscope is inserted,and a biopsy needle and/or a biopsy forceps are extended at the distalend. Thus, a tissue thereof can be sampled.

In a tract inside of the body having multi-level branches such as thebronchi, when a position of an abnormal part is close to the end of thebranch, bringing the distal end of an endoscope to a target partaccurately in a short period of time is difficult. Therefore, forexample, a navigation apparatus is proposed in Japanese UnexaminedPatent Application Publication No. 2000-135215 above.

By the way, for a diagnosis on an internal organ of the abdomen area,which is a body to be examined, an image analysis software isconventionally in actual use which creates a three-dimensional virtualimage of the body to be examined within the abdomen area mainly asdescribed above and displays the image for diagnosis.

An image system using this kind of image analysis software is used fordiagnosis so that a doctor can identify a change in a lesion of a bodyto be examined within the abdomen area of a patient before surgery, andthe diagnosis is generally performed on a desk.

SUMMARY OF THE INVENTION

An object observation system of the invention has an observationapparatus for observing a body to be examined, a three-dimensional imagerecording apparatus for recording three-dimensional images, which areobtained in advance, of the body to be examined, and an imageconstructing apparatus for constructing a three-dimensional image basedon images in synchronization with the observation apparatus, which arerecorded in the three-dimensional image recording apparatus.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an entire configuration diagram showing an endoscopic surgerysystem according to a first embodiment;

FIG. 2 is a circuit block diagram of an endoscope apparatus andrendering apparatus in FIG. 1;

FIG. 3 is a control flowchart of a system controller in FIG. 1;

FIG. 4 is a circuit block diagram of an endoscope apparatus andrendering apparatus included in an endoscopic surgery system accordingto a second embodiment;

FIG. 5 is a control flowchart of a system controller in FIG. 4;

FIG. 6 is a circuit block diagram of an endoscope apparatus andrendering apparatus showing a variation example of FIG. 4;

FIG. 7 is a control flowchart of a system controller showing a variationexample of FIG. 5;

FIG. 8 is a diagram showing a display example of a synthesized image;

FIG. 9 is an entire configuration diagram showing an endoscopic surgerysystem according to a third embodiment;

FIG. 10 is a circuit block diagram of an endoscope apparatus andrendering apparatus in FIG. 9;

FIG. 11 is a flowchart of image processing to be performed by arendering image creating apparatus in FIG. 9;

FIG. 12 is an image display example showing a rendering image of theinside of a body cavity around a target part, which is created by therendering image creating apparatus in FIG. 9;

FIG. 13 is a conceptual diagram showing processing pattern imagesextracted from a three-dimensional image of the inside of the bodycavity in FIG. 12 and a synthesized image created by synthesizing theseprocessing pattern images;

FIG. 14 is an image display example of the synthesized image in FIG. 13after subtraction processing;

FIG. 15 is a conceptual diagram in which a desired rendering image isobtained by directly instructing selective display of the synthesizedimage in FIG. 13;

FIG. 16 is a configuration diagram showing a configuration of a remotesurgery supporting apparatus and surgery system according to a fourthembodiment of the invention;

FIG. 17 is a diagram showing a state in which a rigid endoscope in FIG.16 is being used;

FIG. 18 is a diagram showing a construction of the rigid endoscope inFIG. 17;

FIG. 19 is a diagram showing a construction of an essential part of atrocar in FIG. 17;

FIG. 20 is a first flowchart showing a processing flow of a remotesurgery supporting apparatus and surgery system in FIG. 16;

FIG. 21 is a second flowchart showing a processing flow of a remotesurgery supporting apparatus and surgery system in FIG. 16;

FIG. 22 is a diagram showing a VR display screen to be displayed on a VRimage display monitor in FIG. 16;

FIG. 23 is a diagram showing a support image created by a supportinformation creating apparatus in FIG. 16;

FIG. 24 is a diagram showing a first example of an endoscopic imagedisplayed on an endoscopic image display monitor in FIG. 16;

FIG. 25 is a diagram showing a VR display screen to be displayed inaccordance with the endoscopic image in FIG. 24;

FIG. 26 is a diagram showing a second example of an endoscopic imagedisplayed on the endoscopic image display monitor in FIG. 16;

FIG. 27 is a diagram showing a VR display screen displayed in accordancewith the endoscopic image in FIG. 26;

FIG. 28 is a configuration diagram showing a configuration of a remotesurgery supporting apparatus and surgery system according to a fifthembodiment of the invention;

FIG. 29 is a configuration diagram showing a configuration of a remotesurgery supporting apparatus and surgery system according to a sixthembodiment of the invention;

FIG. 30 is a configuration diagram showing the rigidity of a surgerysupporting apparatus according to a seventh embodiment of the invention;

FIG. 31 is a diagram showing a state in which a rigid endoscope in FIG.30 is being used;

FIG. 32 is a diagram showing a construction of the rigid endoscope inFIG. 31;

FIG. 33 is a diagram showing a construction of an essential part of atrocar in FIG. 31;

FIG. 34 is a top view showing a top face of an XY-inserting pointmeasuring apparatus in FIG. 30;

FIG. 35 is a side view showing a side of the XY-inserting pointmeasuring apparatus in FIG. 30;

FIG. 36 is a back view showing a back face of the XY-inserting pointmeasuring apparatus in FIG. 30;

FIG. 37 is a diagram showing a construction of a Z-inserting pointmeasuring apparatus in FIG. 30;

FIG. 38 is a flowchart showing a processing flow of a surgery supportapparatus in FIG. 30;

FIG. 39 is a flowchart showing a flow of XY-inserting point measuringprocessing in FIG. 38;

FIG. 40 is a first diagram illustrating the XY-inserting point measuringprocessing in FIG. 39;

FIG. 41 is a second diagram illustrating the XY-inserting pointmeasuring processing in FIG. 39;

FIG. 42 is a third diagram illustrating the XY-inserting point measuringprocessing in FIG. 39;

FIG. 43 is a flowchart showing a flow of Z-inserting point measuringprocessing in FIG. 38;

FIG. 44 is a first diagram illustrating the Z-inserting point measuringprocessing in FIG. 43;

FIG. 45 is a second diagram illustrating the Z-inserting point measuringprocessing in FIG. 43;

FIG. 46 is a diagram showing a VR display screen displaying a VR imageconstructed/created by processing in FIG. 39;

FIG. 47 is a diagram showing a first example of an endoscopic imagedisplayed on an endoscopic image display monitor in FIG. 30;

FIG. 48 is a diagram showing a VR display screen displayed in accordancewith the endoscopic image in FIG. 47;

FIG. 49 is a diagram showing a second example of an endoscopic imagedisplayed on the endoscopic image display monitor in FIG. 30;

FIG. 50 is a diagram showing a VR display screen displayed in accordancewith the endoscopic image in FIG. 49;

FIG. 51 is a diagram showing a third example of an endoscopic imagedisplayed on the endoscopic image display monitor in FIG. 30;

FIG. 52 is a diagram showing a VR display screen displayed in accordancewith the endoscopic image in FIG. 51;

FIG. 53 is a diagram showing a VR display screen displaying a VR imagehaving a different scale from that of the VR image in FIG. 52;

FIG. 54 is a construction diagram showing a construction of a techniquesupport system according to an eighth embodiment of the invention;

FIG. 55 is a block diagram showing an essential configuration of thetechnique support system in FIG. 54;

FIG. 56 is a diagram showing a construction of an endoscope in FIG. 54;

FIG. 57 is a diagram illustrating an operation of the technique supportsystem in FIG. 54;

FIG. 58 is a flowchart showing a processing flow of the techniquesupport system in FIG. 54;

FIG. 59 is a first diagram showing a screen developed in the processingin FIG. 58;

FIG. 60 is a second diagram showing a screen developed in the processingin FIG. 58;

FIG. 61 is a third diagram showing a screen developed in the processingin FIG. 58;

FIG. 62 is a first diagram illustrating a variation example of anoperation of the technique support system in FIG. 54;

FIG. 63 is a second diagram illustrating a variation example of anoperation of the technique support system in FIG. 54;

FIG. 64 is a diagram showing a first variation example of a screendeveloped in the processing in FIG. 58;

FIG. 65 is a first diagram showing a second variation example of ascreen developed in the processing in FIG. 58;

FIG. 66 is a second diagram showing a second variation example of ascreen developed in the processing in FIG. 58;

FIG. 67 is a third diagram showing the second variation example of ascreen developed in the processing in FIG. 58;

FIG. 68 is a diagram illustrating a side-view observation direction of aside-view endoscope in FIG. 54;

FIG. 69 is a flowchart showing processing for correcting a generalvirtual image, which is compliant with the side-view endoscope in FIG.68;

FIG. 70 is a first diagram showing a screen developed by the processingin FIG. 69;

FIG. 71 is a second diagram showing a screen developed by the processingin FIG. 69;

FIG. 72 is a construction diagram showing a construction of a techniquesupport system according to a ninth embodiment of the invention;

FIG. 73 is a block diagram showing an essential configuration of thetechnique support system in FIG. 72;

FIG. 74 is a diagram showing a construction of an endoscope in FIG. 72;

FIG. 75 is a diagram illustrating an operation of the technique supportsystem in FIG. 72;

FIG. 76 is a flowchart showing a processing flow of the techniquesupport system in FIG. 72;

FIG. 77 is a first diagram showing a screen developed in the processingin FIG. 76;

FIG. 78 is a second diagram showing a screen developed by the processingin FIG. 76;

FIG. 79 is a first diagram illustrating an operation in which apoint-of-vision information input portion in FIG. 72 is a sensorprovided at a handle of the endoscope;

FIG. 80 is a second diagram illustrating an operation in which thepoint-of-vision information input portion in FIG. 72 is a sensorprovided at the handle of the endoscope;

FIG. 81 is a diagram showing a head band having the point-of-view inputportion in FIG. 72;

FIG. 82 is a diagram showing a state in which the head band in FIG. 81is put on;

FIG. 83 is a schematic construction diagram showing a virtual imagedisplay apparatus according to a tenth embodiment of the invention andshowing an entire construction of an endoscope system including theapparatus;

FIG. 84 is a block diagram showing an entire configuration of theendoscope system in FIG. 83;

FIG. 85 is a perspective view showing an external construction of theendoscope in FIG. 83;

FIG. 86 is a perspective view showing a construction example in whichthe system is attached to the arm of an operator;

FIG. 87 is a perspective view showing an external construction of atrocar, which is an attachment target portion to which a sensor isattached;

FIG. 88 is a construction perspective view showing a first variationexample of the attachment target portion;

FIG. 89 is a construction perspective view showing a second variationexample of the attaching target portion;

FIG. 90 is a diagram illustrating a display operation of this embodimentand showing a display example of an operator monitor shown in FIG. 83;

FIG. 91 is a flowchart illustrating a display operation of thisembodiment and showing main control processing by a CPU of a virtualimage creating section;

FIG. 92 is a schematic construction diagram showing a virtual imagedisplay apparatus according to an eleventh embodiment of the inventionand showing an entire construction of an endoscope system including theapparatus;

FIG. 93 is a block diagram showing an entire configuration of theendoscope system in FIG. 92;

FIG. 94 is a perspective view showing an external construction of atrocar, which is an attachment target portion to which a sensor isattached;

FIG. 95 is a flowchart illustrating a display operation of the eleventhembodiment and showing main control processing by a CPU of a virtualimage creating section;

FIG. 96 is a flowchart illustrating a display operation of the eleventhembodiment and showing voice control processing by the CPU;

FIG. 97 relates to a virtual image display apparatus of a twelfthembodiment and is a flowchart showing main control processing by a CPUof a virtual image creating section;

FIG. 98 is an entire configuration diagram showing an object observationsystem according to a thirteenth embodiment;

FIG. 99 is a construction diagram showing a construction of a remotecontroller for an operator in FIG. 98;

FIG. 100 is a screen display example of a virtual image display screendisplayed on a VR monitor in FIG. 98;

FIG. 101 is a screen display example on which a virtual image isdisplayed in a virtual image display area in FIG. 100;

FIG. 102 is an example of an endoscopic live image displayed on anendoscope monitor in FIG. 98;

FIG. 103 is an example of an endoscopic live image displayed on theendoscope monitor when the endoscope is moved;

FIG. 104 is a screen display example in which a virtual image agreeingwith the endoscopic live image in FIG. 103 is displayed on the virtualimage display area;

FIG. 105 is a flowchart showing a processing operation, which is afeature of the thirteenth embodiment;

FIG. 106 is an example of an endoscopic live image for illustrating anoperation of the thirteenth embodiment;

FIG. 107 is a first screen display example of a virtual image displayscreen for illustrating the operation of the thirteenth embodiment;

FIG. 108 is a screen display example of a virtual image display screenon which a virtual image in FIG. 107 is enlarged;

FIG. 109 is a second screen display example of a virtual image displayscreen for illustrating an operation of the thirteen embodiment;

FIG. 110 is a screen display example of a virtual image display screenwhen organ removal processing is performed on the virtual image in FIG.108;

FIG. 111 is an entire configuration diagram of an object observationsystem showing a variation example of the thirteenth embodiment;

FIG. 112 is an entire configuration diagram showing an objectobservation system according to a fourteenth embodiment;

FIG. 113 is a flowchart showing processing operation, which is a featureof the fourteenth embodiment;

FIG. 114 is an entire configuration diagram showing an objectobservation system of a fifteenth embodiment;

FIG. 115 is a construction diagram showing a construction of anoperator's remote controller in FIG. 114;

FIG. 116 is a screen display example of a virtual image display screenin a three-dimensional display form, which is displayed on a VR monitorin FIG. 114;

FIG. 117 is a screen display example on which a virtual image isdisplayed in a virtual image display area in FIG. 116;

FIG. 118 is a screen display example of a virtual image display screenin a two-dimensional display form, which is displayed on the VR monitorin FIG. 114;

FIG. 119 is a screen display example of an equipment setting informationscreen displayed on the VR monitor in FIG. 114;

FIG. 120 is a flowchart showing a processing operation, which is afeature of the fifteenth embodiment;

FIG. 121 is a screen display example of a virtual image display screenfor illustrating an operation of the fifteenth embodiment;

FIG. 122 is a screen display example of a virtual image display screenon which the virtual image in FIG. 121 is enlarged;

FIG. 123 is an entire configuration diagram showing an objectobservation system according to a sixteenth embodiment; and

FIG. 124 is a flowchart showing a processing operation, which is afeature of the sixteenth embodiment.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS First Embodiment

An embodiment of the invention will be described below with reference todrawings.

First Embodiment

FIGS. 1 to 3 relate to a first embodiment of the invention. FIG. 1 is anentire configuration diagram showing an endoscopic surgery systemaccording to the first embodiment. FIG. 2 is a circuit block diagram ofan endoscope apparatus and rendering apparatus in FIG. 1. FIG. 3 is acontrol flowchart of a system controller in FIG. 1.

As shown in FIG. 1, an endoscopic surgery system 1 according to thefirst embodiment is an object observation system. The endoscopic surgerysystem 1 has a rigid endoscope (often simply called endoscope,hereinafter), which is inserted to an abdominal cavity of a patient 3,which is a body to be examined, lying on an operation table 2 through atrocar (not shown). A TV camera head 4 self-containing an image pickupapparatus is attached to the rigid endoscope 5. The endoscopic surgerysystem 1 includes a pneumoperitoneum guide tube 6 and an electric knifeprobe 7, which are inserted to the patient 3. The pneumoperitoneum guidetube 6 is used for performing a pneumoperitoneum. The electric knifeprobe 7 is used for electrically performing a cautery treatment on anaffected part.

In the endoscopic surgery system 1, a signal cable 8 connecting to theTV camera head 4, a light guide cable 9 connecting to the endoscope 5, apneumoperitoneum tube 10 connecting to the pneumoperitoneum guide tube 6and a signal cable 11 connecting to the electric knife probe 7 areconnected to a CCU 13, a light source apparatus (which may be calledlight source hereinafter) 14, a pneumoperitoneum apparatus 15, and anelectric knife 16, which are mounted in a trolley 12, respectively.

A system controller 17, a VTR 18 and an endoscope monitor 19 are mountedin the trolley 12 in addition to the CCU 13, the light source 14, andthe pneumoperitoneum apparatus 15 and the electric knife 16. The CCU 13performs signal processing for an image pickup apparatus contained inthe TV camera head 4. The light source 14 supplies illumination light.The pneumoperitoneum apparatus 15 supplies gas for pneumoperitoneum. Theelectric knife 16 supplies cautery high frequency power. The systemcontroller 17 performs entire control. The VTR 18 records image signalsoutput from the CCU 13. The endoscope monitor 19 displays image signalsoutput from the CCU 13 as endoscopic images.

In the endoscopic surgery system 1, a central operation panel 21 forperforming central operation and a central display panel 22 forperforming central display are attached to the trolley 12. A remotecontroller 23 for performing remote control operation is removablyprovided in the operation table 2.

Medical equipment such as the CCU 13 are connected to the systemcontroller 17 through a communication cable (not shown) and arecentrally operated by the central operation panel 21, the remotecontroller 23 and the central display panel 22.

The system controller 17 has a microphone 24 for capturing voice asinstructing means. The microphone 24 is removably connected to thesystem controller 17 through a signal cable extending from the head set25. The microphone 24 may be a pin microphone. The system controller 17and the head set 25 may be adjusted to communicate voice information byradio communication such as infrared rays. The microphone 24 may beattached to a goggles type or glasses type apparatus called face mountdisplay (FMD) or head mount display (HMD).

A foot switch 26, which is a remote operation unit, is connected to thesystem controller 17. A hand switch (not shown) may be connected to thesystem controller 17 instead of the foot switch 26.

The system controller 17 receives image signals output from the CCU 13and causes the VTR 18 to record the image signals. When a release switch(not shown) is manipulated, the system controller 17 receives andrecords still-image data from the CCU 13.

The endoscopic surgery system 1 according to this embodiment includes arendering apparatus 28 which can create and display a rendering image asa virtual three-dimensional image of the inside of a body cavity byusing medical image data in a three-dimensional area. The renderingapparatus 28 is included in a three-dimensional image recorder in whichthree-dimensional images, which has been acquired in advance, of a bodyto be examined are recorded.

As shown in FIG. 2, the endoscope 5, the TV camera head 4, the CCU 13and the endoscope monitor 19 are included in the endoscope apparatus 29as an observation apparatus for observing a body to be examined. Theendoscope 5 has a sensor 31 as a positional relationship detectingportion for detecting a relative positional relationship between thedistal end part of the insert portion and a body to be examined,positional relationship information with respect to a twisting angle,insert length, and insert point and focus point with respect to a bodyto be examined of the insert portion can be detected. According to thisembodiment, a rigid endoscope is used. However, when a soft endoscopehaving a soft insert portion is used, an inserting speed of the insertportion, a bending angle of a bend and so on can be detected as thepositional relationship information.

Positional relationship information detected by the sensor 31 iscaptured by the system controller 17 and is output to the renderingapparatus 28. The sensor 31 may perform radio communication by infraredrays, for example, and may give the positional relationship informationdirectly to the rendering apparatus 28.

The rendering apparatus 28 reconstructs and displays on the renderingmonitor 27 a body cavity rendering image following the distal end of theinsert portion of the endoscope 5, that is, in synchronization with anendoscopic image displayed on the endoscope monitor 19 based on theposition relationship information obtained from the sensor 31.

The rendering apparatus 28 has a rendering image creating apparatus 28A,an operation instructing section 32 such as a mouse and a keyboard, anda pattern image storing section 33 for storing extracted image, which iscreated by instructing to set a predetermined parameter relating toimage display in response to an instruction from the operationinstructing section 32 and extracting a predetermined part.

The endoscope apparatus 29 has a release switch (not shown) in the TVcamera head 4 or the CCU 13. By manipulating the release switch,photo-shooting is performed, and still image data of endoscopic imagesis recorded as records of surgery. When the endoscope is an electronicendoscope self-containing an image pickup apparatus, the release switchis provided in the electronic endoscope.

In order to recognize details of surgery easily, the endoscopic surgerysystem 1 according to this embodiment is adjusted to store still imagedata of endoscopic images and rendering image data substantially at thesame time in response to a manipulation on the release switch.

More specifically, the system controller 17 includes an endoscopic imagestoring section 41 as recording means for recording still image data ofendoscopic images in response to a release signal output from the CCU13. The rendering image creating apparatus 28A includes a renderingimage storing section 42 as recording means for recording renderingimage data in response to a release signal output from the CCU 13.

The rendering image creating apparatus 28A associates still image dataof an endoscopic image stored in the endoscopic image storing section 41and rendering image data stored in the rendering image storing section42 by using a time stamp, for example.

Therefore, the endoscopic surgery system 1 according to this embodimentstores rendering image data in association with still image data of anendoscopic image substantially in synchronization with the still imagedata of the endoscopic image in response to a manipulation on therelease switch. The endoscopic image storing section 41 and therendering image storing section 42 may not be provided separately likethe pattern image storing section 33.

The endoscopic surgery system 1 having the above-described constructionhas a construction as illustrated in FIG. 1 and may be used for anendoscopic surgery.

The insert portion of the endoscope 5 is inserted to a body cavity of apatient, and an endoscopic image obtained by the endoscope 5 is pickedup by the TV camera head 4. The TV camera head 4 picks up andoptoelectronically converts an endoscopic image and outputs image pickupsignals thereof to the CCU 13. The CCU 13 performs signal processing onthe image pickup signals and generates image signals.

On the other hand, the rendering image creating apparatus 28Areconstructs a body-cavity rendering image in accordance with the distalend of the insert portion of the endoscope 5, that is, insynchronization with an endoscopic image displayed on the endoscopemonitor 19 based on positional relationship information obtained fromthe sensor 31.

Here, the CCU 13 and the rendering image creating apparatus 28A arecontrolled by the system controller 17, and processing is performed inaccordance with the flowchart shown in FIG. 3.

As shown in FIG. 3, the system controller 17 controls the CCU 13 tooutput endoscopic image signals created by the CCU 13 to the endoscopemonitor 19 and display the endoscopic image on a display screen of theendoscope monitor 19. Furthermore, the system controller 17 controls therendering image creating apparatus 28A to output rendering image datacreated by the rendering image creating apparatus 28A to the renderingmonitor 27 and displays the rendering image on a display screen of therendering monitor 27 (step S1).

An operator uses an electric knife 16, for example, to performtreatments with reference to endoscopic images and rendering images.

Here, an operator manipulates the release switch in the TV camera head 4or the CCU 13 to take photographs and records still image data ofendoscopic images as records of surgery.

The system controller 17 judges the presence of a release signal (stepS2). If a release signal is given, still image data of endoscopic imagesis recorded in the endoscopic image storing section 41 (step S3).

Next, the system controller 17 controls the rendering image creatingapparatus 28A to record rendering image data associated with the stillimage data of an endoscopic image in the rendering image storing section42 (step S4). If a release signal is not given or if surgery ends, thesystem controller 17 terminates the processing. In this way, the systemcontroller 17 constructs a rendering image from images recorded in therendering apparatus 28 in synchronization with a still image of anendoscopic image.

The still image data and rendering image data of an endoscopic image maybe recorded in reverse order. In other words, rendering image dataassociated with the still image data of an endoscopic image may berecorded in the rendering image storing section 42 first, and the stillimage data of the endoscopic image may be then recorded in theendoscopic image storing section 41.

Thus, the endoscopic surgery system 1 according to this embodiment canrecord rendering image data in association with the still image data ofan endoscopic image substantially at the same time and can attach therendering image data to patient's charts along with the still image dataof the endoscopic image.

Here, rendering image data does not have the unnecessary blood, fat andso on. Thus, viewing the rendering image data along with the recordedstill image data of an endoscopic image helps recognizing whichtechnique step of what surgery the endoscope image relates to.

Therefore, with the endoscopic surgery system 1 according to thisembodiment, details of surgery can be grasped easily.

The endoscopic surgery system 1 according to this embodiment includes asensor 31 as a positional relationship detecting portion for detecting arelative positional relationship between the distal end part of theinsert portion of the endoscope 5 and a body to be examined, positionalrelationship information with respect to a twisting angle, insertlength, and insert point and focus point with respect to a body to beexamined of the insert portion can be detected. However, the inventionis not limited thereto. A relative positional relationship between thedistal end of the insert portion of the endoscope 5 and a body to beexamined may be detected by a positional relationship detecting portionby detecting a position and/or angle of a body-cavity tissue throughimage processing on an endoscopic image.

Second Embodiment

FIGS. 4 to 8 relate to a second embodiment of the invention. FIG. 4 is acircuit block diagram of an endoscope apparatus and rendering apparatusincluded in an endoscopic surgery system according to the secondembodiment. FIG. 5 is a control flowchart of a system controller in FIG.4. FIG. 6 is a circuit block diagram of an endoscope apparatus andrendering apparatus showing a variation example of FIG. 4. FIG. 7 is acontrol flowchart of a system controller showing a variation example ofFIG. 5. FIG. 8 is a display example of a synthesized image.

While, according to the first embodiment, rendering image data and thestill image data of an endoscopic image are associated and are recordedin separate storing portions, rendering image data and the still imagedata of an endoscopic image are recorded in a same storing portionaccording to the second embodiment. Since the other construction is thesame as the one according to the first embodiment, the descriptionsthereof will be omitted here. The same reference numerals are given tothe same components for description.

In other words, as shown in FIG. 4, an endoscopic surgery systemaccording to the second embodiment records still image data of anendoscopic image and rendering image data thereof in a same storingportion. More specifically, a rendering image creating apparatus 28Bincludes an image storing section 43 as recording means for recordingstill image data of an endoscopic image and rendering image datathereof.

Then, still image data of an endoscopic image is output to the renderingimage creating apparatus 28B through the system controller 17B inresponse to a release signal output from the CCU 13. The rendering imagedata is recorded in the image storing section 43 along with the stillimage data of the endoscopic image. As described later, the imagestoring section 43 records the still image data of the endoscopic imageand the rendering image data in a same folder simultaneously.

The endoscopic surgery system having the above-described constructionhas the same construction as that of the first embodiment and can beapplied for an endoscopic surgery.

The insert portion of the endoscope 5 is inserted to a body cavity of apatient, and an endoscopic image obtained by the endoscope 5 is pickedup by the TV camera head 4. The TV camera head 4 picks up andoptoelectronically converts an endoscopic image and outputs image pickupsignals thereof to the CCU 13. The CCU 13 performs signal processing onthe image pickup signals and generates image signals.

On the other hand, a rendering image creating apparatus 28B reconstructsa body-cavity rendering image in accordance with the distal end of theinsert portion of the endoscope 5, that is, in synchronization with anendoscopic image displayed on the endoscope monitor 19 based onpositional relationship information obtained from the sensor 31.

Here, the CCU 13 and the rendering image creating apparatus 28B arecontrolled by the system controller 17B, and processing is performed inaccordance with the flowchart shown in FIG. 5.

As shown in FIG. 5, the system controller 17B controls the CCU 13 tooutput endoscopic image signals created by the CCU 13 to the endoscopemonitor 19 and display the endoscopic image on a display screen of theendoscope monitor 19. Furthermore, the system controller 17B controlsthe rendering image creating apparatus 28B to output rendering imagedata created by the rendering image creating apparatus 28B to therendering monitor 27 and display the rendering image on a display screenof the rendering monitor 27 (step S11).

An operator uses an electric knife 16, for example, to performtreatments with reference to endoscopic images and rendering images.

Here, in order to record still image data of endoscopic images asrecords of surgery, an operator manipulates the release switch in the TVcamera head 4 or the CCU 13 to take photographs.

Then, the system controller 17B judges the presence of a release signal(step S12). If a release signal is given, the saving folder isidentified (step S13). Then, still image data of endoscopic images isoutput to the rendering image creating apparatus 28B and record thestill image data of the endoscopic images (step S14).

Next, the system controller 17B records rendering image data inassociation with the still image data of an endoscopic image in theimage storing section 43 (step S15). If a release signal is not given orif surgery ends, the system controller 17B terminates the processing.

Thus, in addition to the same advantages as those of the firstembodiment, the endoscopic surgery system according to the secondembodiment can obtain an advantage that still image data of anendoscopic image and rendering image data thereof can be searched easilywithout consideration of a combination of the still image data of theendoscopic image and the rendering image data. This is because the stillimage data of the endoscopic image and the rendering image data arerecorded simultaneously.

The image storing section 43 may be included in a system controller 17Cas shown in FIG. 6 instead of providing in the rendering image creatingapparatus 28B.

In this case, the system controller 17C records rendering image dataoutput from a rendering image creating apparatus 28C in response to arelease signal in the image storing section 43 in synchronization withthe still image data of an endoscopic image.

A control flow of the system controller 17B may have a construction asshown in FIG. 7.

As shown in FIG. 7, the system controller 17B controls the CCU 13 tooutput endoscopic image signals created by the CCU 13 to the endoscopemonitor 19 and display the endoscopic image on a display screen of theendoscope monitor 19. Furthermore, the system controller 17B controlsthe rendering image creating apparatus 28B to output rendering imagedata created by the rendering image creating apparatus 28B to therendering monitor 27 and display the rendering image on a display screenof the rendering monitor 27 (step S21).

Then, an operator uses the electric knife 16, for example, to performtreatments with reference to endoscopic images and rendering images.

Here, an operator manipulates the release switch in the TV camera head 4or the CCU 13 to take photographs and record still image data ofendoscopic images as records of surgery.

The system controller 17B judges the presence of a release signal (stepS22). If a release signal is given, the system controller 17B causesstill image data of endoscopic images to output to the rendering imagecreating apparatus 28B and synthesizes the still image data of theendoscopic image and the rendering image data. As a result, asynthesized image thereof is created (step S23).

Next, the system controller 17B records the synthesized image data inthe image storing section 43 (step S24).

Here, a synthesized image is one image in which an endoscopic image(still image) and a rendering image are placed in parallel, for example,as shown in FIG. 8. In a synthesized image, an endoscopic image (stillimage) and a rendering image may be placed vertically instead of placingin parallel. Alternatively, a synthesized image may be displayed on asub-screen with respect to an endoscopic image (still image) in P-in-Pdisplay form.

If a release signal is not given or if surgery ends, the systemcontroller 17 terminates the processing.

Thus, in addition to the same advantages as those of the secondembodiment, the endoscopic surgery system in this variation example canobtain an advantage that still image data of an endoscopic image andrendering image data thereof can be searched easily withoutconsideration of a combination of the still image data of the endoscopicimage and the rendering image data. This is because the synthesizedimage of the still image data of the endoscopic image and the renderingimage data are recorded.

An endoscopic surgery system of this embodiment has an advantage thatdetails of surgery can be grasped easily.

Third Embodiment

FIGS. 9 to 15 relate to the third embodiment of the invention. FIG. 9 isan entire configuration diagram showing an endoscopic surgery system.FIG. 10 is a circuit block diagram of an endoscope apparatus andrendering apparatus in FIG. 9. FIG. 11 is a flowchart of imageprocessing to be performed by a rendering image creating apparatus inFIG. 9. FIG. 12 is an image display example showing a rendering image ofthe inside of a body cavity around a target part, which is created bythe rendering image creating apparatus in FIG. 9. FIG. 13 is aconceptual diagram showing processing pattern images extracted from arendering image of the inside of the body cavity in FIG. 12 and asynthesized image created by synthesizing these processing patternimages. FIG. 14 is an image display example of the synthesized image inFIG. 13 after subtraction processing. FIG. 15 is a conceptual diagram inwhich a desired rendering image is obtained by directly instructingselective display of the synthesized image in FIG. 13.

The same reference numerals are given to the same components as those ofthe first embodiment. The descriptions will be omitted herein, anddifferent components and operations will be mainly described below.

Conventionally, based on an instruction by an operator in a clean area,a nurse or operator in an unclean area manipulates a keyboard, forexample, and causes a rendering image to be displayed as a referenceimage.

With an endoscopic surgery system 1A of this embodiment, an operator cangive an instruction by voice directly through a microphone 24 and easilyoperate. Thus, a desired rendering image of a surrounding of a targetpart can be obtained. An operator may give instructions by using notonly the microphone 24 but also a mouse and/or keyboard or a remotecontroller.

Image data of a rendering image is output to a switcher 142 through asplitter 141, is switched by the switcher 142 from peripheral equipmentinformation from a system controller 17 and is output to a display panel22.

As described above, a rendering image creating apparatus 28A creates anextracted image by extracting a predetermined part from anin-body-cavity rendering image around a target part when a predeterminedparameter relating to image display is instructed to set in response toan instruction of the operation instructing section 32. The renderingimage creating apparatus 28A outputs the created extracted image data toa pattern image storing section 33 and causes the pattern image storingsection 33 to store the created extracted image data.

Further describing the extracting processing, the rendering apparatus 28creates multiple processing pattern images as shown in Table 1, forexample, by performing extracting processing in response to a settinginstruction by the operation instructing section 32 in advance withrespect to an in-body-cavity rendering image around a target part. TABLE1 PROCESSING PARAMATERS PATERN AMBIENT DIFFUSE SPECULAR LIGHT IMAGESLIGHT LIGHT LIGHT STRENGTH TRANSPARENCY CLEARNESSTARGET * * * * * * * * * * * * * * * * * * ORGAN IMAGE IMAGEOF * * * * * * * * * * * * * * * * * * TARGET ORGAN BLOOD VESSELS IMAGEOF * * * * * * * * * * * * * * * * * * PART BEFORE TARGET ORGAN

Table 1 shows processing patterns for three images including an imagebefore a target organ, an image of blood vessels of the target organ andan image of the target organ as processing pattern images for a targetorgan in accordance with progress of surgery.

Parameters shown in Table 1 include ambient light, diffuse light,specular light, light strength, transparency and clearness. Theprocessing pattern images, which will be described later, are definedbased on these parameters.

Here, ambient light refers to light in the environment. Diffuse lightrefers to scattered light. Specular light refers to light havingreflected waves traveling in a constant direction on a diffusivereflecting surface. Clearness refers to contrast at the edges of animage. The parameters may further include light attenuation and angle ofview.

The rendering image creating apparatus 28A performs synthesizingprocessing on multiple extracted processing pattern images and creates asynthesized image. In other words, the rendering image creatingapparatus includes an image extracting processing portion and asynthesizing processing portion. The rendering image creating apparatus28A may be constructed so as to perform subtraction processing on asynthesized image and create subtraction-processed image.

As an in-body-cavity rendering image a target part and surroundings, therendering apparatus 28 creates a synthesized image from the processingpattern images read from the pattern image storing section 33 inaccordance with a voice instruction by an operator from the microphone24 to the rendering image creating apparatus 28A through the systemcontroller 17 and displays a desired rendering image on the renderingmonitor 27. The microphone 24 may be constructed so as to communicatevoice information by radio communication with infrared rays, forexample, and may give a voice instruction directly to the renderingimage creating apparatus 28A.

The endoscopic surgery system 1A having the above-described constructionmay have the construction as described with reference to FIG. 9 and canbe applied to an endoscopic surgery.

Here, the rendering image creating apparatus 28A performs imageprocessing based on the flowchart shown in FIG. 11.

First of all, before surgery, the rendering image creating apparatus 28Aperforms extraction processing in response to a manipulation on theoperation instructing section 32 by an operator, a nurse or an operatorand in accordance with the parameters described with reference to Table1 from in-body-cavity rendering image of a target part and surroundingsshown in FIG. 12 (step S31). Then, the rendering image creatingapparatus 28A creates a processing pattern image as shown in FIG. 13.Image processing is performed on blood vessels such that the bloodvessels can be seen through organs therebefore.

Here, as processing pattern images, an image before a target organ, animage of blood vessels of the target organ and an image of the targetorgan are created in accordance with the parameters defined in Table 1.

The created processing pattern images are stored in the pattern imagestoring section 33.

The operations up to this point are included in a preparation stagebefore an endoscopic surgery.

Then, an operator advances to an endoscopic surgery.

The insert portion of the endoscope 5 is inserted to a body cavity of apatient, and an endoscopic image obtained by the endoscope 5 is pickedup by the TV camera head 4. The TV camera head 4 picks up andoptoelectronically converts an endoscopic image and outputs image pickupsignals thereof to the CCU 13. The CCU 13 performs signal processing onthe image pickup signals and generates image signals. The CCU 13 outputsthe image signals to the endoscope monitor 19 and causes the endoscopicimage to be displayed on the endoscope monitor 19.

An operator uses an electric knife 16, for example, to performtreatments with reference to the endoscopic images and rendering images.

Here, a rendering image creating apparatus 28A reconstructs abody-cavity rendering image in accordance with the distal end of theinsert portion of the endoscope 5, that is, in synchronization with anendoscopic image displayed on the endoscope monitor 19 based onpositional relationship information obtained from the sensor 31. Then,the rendering image creating apparatus 28A displays the body-cavityrendering image on the rendering monitor 27.

Here, an operator instructs to “synthesize images” through themicrophone 24 with respect to a body-cavity rendering image (refer toFIG. 12) of a target part and surroundings displayed on the renderingmonitor 27.

Thus, the rendering image creating apparatus 28A judges whether a voiceinstruction from the microphone 24 is given or not (step S32). If thevoice instruction is “synthesize images”, the three processing patternimages are read out from the pattern image storing section 33 and thethree processing pattern images are synthesized as shown in FIG. 13(step S33). The resulting synthesized image is displayed on therendering monitor 27.

Then, in accordance with progress of the surgery, the operator gives avoice instruction from the microphone 24 and displays a desiredrendering image on the rendering monitor 27 with respect to thesynthesized image.

Here, the operator performs treatments on a target organ by using anelectric knife, for example, with reference to the endoscopic images andthe rendering images. Then, the rendering image creating apparatus 28Arepeats the steps S32 and S33 in accordance with a next voiceinstruction until the operator instructs to finish.

Thus, the operator can refer to a desired rendering image and can checka target organ from the desired rendering image when an endoscopic imageis not clear enough to view.

Therefore, the endoscopic surgery system 1A of this embodiment can beeasily operated, and a desired rendering image can be obtained.

The rendering image creating apparatus 28A may perform subtractionprocessing as shown in FIG. 14 as synthesizing processing.

More specifically, in response to a voice instruction, “part beforetarget organ, delete” by the operator, the rendering image creatingapparatus 28A subtracts an image of a part before the target organ fromthe synthesized image and displays an image having blood vessels in thetarget organ on the rendering monitor 27.

In response to a voice instruction, “blood vessels in the target organ,delete” by the operator, the rendering image creating apparatus 28Asubtracts an image of the target organ blood vessels from the image ofthe blood vessels in the target organ and displays a target organ imageof the target organ only on the rendering monitor 27.

Thus, the operator can refer to rendering images in accordance withprogress of the surgery and can check the target organ from therendering images when an endoscopic image is not clear enough to view.

The endoscopic surgery system 1A may obtain a desired rendering imagewith respect to a synthesized image by directly instructing selectivedisplay as shown in FIG. 15 regardless of progress of the surgery. Morespecifically, in response to a voice instruction, “target organ image”by an operator, the rendering image creating apparatus 28A reads out“target organ image” data from processing pattern images stored in thepattern image storing section 33 and directly switches from asynthesized image to the target organ image.

Thus, the operator can obtain a desired rendering image directlyregardless of progress of surgery.

An endoscopic surgery system according to this embodiment has anadvantage that the endoscopic surgery system can be easily operated, anda desired rendering image can be obtained.

Fourth Embodiment

A fourth embodiment of the invention will be described below withreference to drawings.

FIGS. 16 to 27 relate to the fourth embodiment of the invention. FIG. 16is a configuration diagram showing a configuration of a remote surgerysupporting apparatus and surgery system. FIG. 17 is a diagram showing astate in which a rigid endoscope in FIG. 16 is being used. FIG. 18 is adiagram showing a construction of the rigid endoscope in FIG. 17. FIG.19 is a diagram showing a construction of an essential part of a trocarin FIG. 17. FIG. 20 is a first flowchart showing a processing flow of aremote surgery supporting apparatus and surgery system in FIG. 16. FIG.21 is a second flowchart showing a processing flow of a remote surgerysupporting apparatus and surgery system in FIG. 16. FIG. 22 is a diagramshowing a VR display screen to be displayed on a VR image displaymonitor in FIG. 16. FIG. 23 is a diagram showing a support image createdby a support information creating apparatus in FIG. 16. FIG. 24 is adiagram showing a first example of an endoscopic image displayed on anendoscopic image display monitor in FIG. 16. FIG. 25 is a diagramshowing a VR display screen to be displayed in accordance with theendoscopic image in FIG. 24. FIG. 26 is a diagram showing a secondexample of an endoscopic image displayed on the endoscopic image displaymonitor in FIG. 16. FIG. 27 is a diagram showing a VR display screendisplayed in accordance with the endoscopic image in FIG. 26.

As shown in FIG. 16, a remote surgery supporting apparatus 201 accordingto this embodiment is disposed in a support room away from surgery roomand remotely supports surgery system 202 disposed in the operation roomthrough a communications line 300.

The surgery system 202 includes, in an operation room, a rigid endoscope203, a system controller 204, a CCU 205, a light source apparatus 206, apneumoperitoneum apparatus 207, an electric knife 208, an ultrasonicprocessor 209, a VTR 210 and a support information player 218. Theremote surgery supporting apparatus 201 includes, outside of theoperation room, a VR image creating apparatus 219 and a supportinformation creating apparatus 220. The surgery system 202 and theremote surgery supporting apparatus 201 are connected through thecommunications line 300.

First of all, details of the surgery system 202 will be described. Imagepickup signals picked up by an image pickup section 211 of the rigidendoscope 203 are transmitted to a CCU 205, undergo image processing andare output to the VTR 210 for recording images and the system controller204.

The system controller 204 includes a communication I/F section 212, amemory 213, a display I/F section 215 and a CPU 216. The communicationI/F section 212 exchanges setting information with the CCU 205, thelight source apparatus 206, the pneumoperitoneum apparatus 207, anelectric knife 208, an ultrasonic treatment apparatus 209, and the VTR210. The memory 213 stores different kinds of programs. The display I/Fsection 215 causes an endoscopic image display monitor 214 to displayimage signals from the CCU 205. The CPU 216 controls these portions.

A remote controller 217 is connected to the CPU 216 of the systemcontroller 204 through the communication I/F section 212. Various kindsof data can be input through the remote controller 217.

The rigid endoscope 203 includes an amount-of-insertion detectingsection 221 and an inclination angle sensor 222. The amount-of-insertiondetecting section 221 detects an inserting amount of the rigid endoscope203. The inclination angle sensor 222 detects an inclination angle ofinsertion of the rigid endoscope 203. Insert amount data detected by theinsert amount detecting portion 221 and insertion inclination angle datadetected by the inclination angle sensor 222 are input to the CPU 216through the communication I/F section 212 of the system controller 204.The CPU 216 outputs the inserting amount data and the insertioninclination angle data to the information transfer I/F section 224through the communication I/F portion 212. The inserting amount data andthe insertion inclination angle data are transmitted by the informationtransfer I/F section 224 to the information transfer I/F section 225 ofthe remote surgery supporting apparatus 1 through the communicationsline 300.

The support information player 218 plays support information includingsupport image information and support voice information from the supportinformation creating apparatus 220 of the remote surgery supportingapparatus 201, which are input from the information transfer I/F section224 through the information transfer I/F section 225 of the remotesurgery supporting apparatus 201 and the communications line 300. Thesupport information player 218 includes a video I/F section 231, adisplay I/F section 233 and a voice I/F section 235. The video I/Fsection 231 inputs support image information. The display I/F section233 displays on the monitor 232 support images including (endoscopicimages+instruction information) based on image information input by thevideo I/F section 231. The voice I/F portion 235 inputs support voiceinformation and causes the speaker 234 to play the support voiceinformation.

As shown in FIG. 17, the rigid endoscope 203 is inserted into the bodyof a patient 239 through trocars 236 and 237 along with a treatingapparatus 238 such as the electric knife 208 and the ultrasonic treatingapparatus 209.

As shown in FIG. 18, the rigid endoscope 203 includes an image pickupsection 211 at the inserted proximal end and the inclination anglesensor 222 at a handle 241 on the inserted proximal end side. Theinclination angle sensor 222 measures an insertion inclination angle ofthe rigid endoscope 203 by using a gyroscopic compass and outputs theresult to the system controller 204.

As shown in FIG. 19, the amount-of-insertion detecting section 221 isprovided on the proximal end side of the trocar 236 for guiding aninsert portion 242 of the rigid endoscope 203 into the body of thepatient 239. The amount-of-insertion detecting section 221 includes aroller 243 and a rotary encoder 244. The roller 243 is in contact withan outer surface of the insert portion 242 and rotates in accordancewith the insertion of the insert portion 242. The rotary encoder 244detects an amount of rotation of the roller 243 and outputs the amountof rotation to the system controller 204 as an amount of insertion ofthe insert portion 242.

Next, details of the remote surgery supporting apparatus 201 will bedescribed. The VR image creating apparatus 219 obtains inserting amountdata and insertion inclination angle data of the rigid endoscope 202from the surgery system 202 through the communications line 300 in realtime. Based on the inserting amount data, the insertion inclinationangle data and CT images obtained by a CT apparatus (not shown), the VRimage creating apparatus 219 creates a volume rendering image (VRimage), which is a virtual image in real time and in a same direction ofline of vision as that of an endoscopic image picked up by the rigidendoscope 202. The support information creating apparatus 220 createssupport image to be transmitted to the surgery system 202 with referenceto VR images thereof.

More specifically, as shown in FIG. 16, the VR image creating apparatus219 includes a recording portion 251, a memory 252, a communication I/Fsection 253, a VR image constructing section 254, a display I/F section256 and a CPU 257. The recording portion 251 stores a CT image database(DB) including multiple CT images. The memory 252 stores different kindsof programs. The communication I/F section 253 receives inserting amountdata detected by the amount-of-insertion detecting section 221 andinsertion inclination angle data detected by the inclination anglesensor 222, from the surgery system 202 through the information transferI/F section 225. The VR image constructing section 254 constructs a VRimage based on inserting amount data and insertion inclination angledata obtained by the communication I/F section 253 and a CT image in theCT image DB. The display I/F section 256 causes the VR image displaymonitor 255 to display a VR image constructed by the VR imageconstructing section 254. The CPU 257 controls these portions. Akeyboard 258 and a mouse 259 used for inputting various kinds of dataare connected to the CPU 257.

The support information creating apparatus 220 includes a video I/Fsection 261, an endoscopic image input section 262, an arrow imageconstructing section 263, an image synthesizing section 264, acommunication I/F section 266, a voice I/F section 268, a display I/Fsection 270, a memory 271, and a CPU 272. The video I/F section 261receives an endoscopic image from the CCU 205 through the informationtransfer I/F sections 224 and 225 and the communications line 300. Theendoscopic image input section 262 converts an endoscopic image obtainedby the video I/F section 261 to digital endoscopic image data. The arrowimage constructing section 263 constructs an arrow image to besuperposed on endoscopic image data. The image synthesizing section 264creates a synthesized image by superposing an arrow image from the arrowimage constructing section 263 on endoscopic image data from theendoscopic image input section 262. The communication I/F section 266receives instruction information from an instruction information inputsection 265 for inputting position information of an arrow image to besuperposed on endoscopic image data. The voice I/F section 268 is usedfor inputting voice data from a microphone 267 used for inputtinginstruction voice. The display I/F section 270 is used for displaying asupport image including (endoscopic image+instruction information),which is a synthesized image from the image synthesizing section 264, ona monitor 269. The memory 271 stores different kinds of programs. TheCPU 272 controls these portions. The voice I/F section 268 and thedisplay I/F section 270 output voice data and support image,respectively, to the support information player 218 of the surgerysystem 202 through the information transfer I/F sections 224 and 225 andthe communications line 300.

An operation of this embodiment having this construction will bedescribed. As shown in FIGS. 20 and 21, at a step S41, the CPU 216 ofthe system controller 204 inputs coordinates of an inserting point (ofan endoscope) where the rigid endoscope 203 is inserted into the body ofthe patient 239 by using the remote controller 217 connecting to thesystem controller 204 of the surgery system 202. The coordinates systemagrees with the coordinate system of the CT image.

At a step S42, the CPU 216 measures and inputs insertion inclinationangle data of the rigid endoscope 203 by using the inclination anglesensor 222. At a step S43, the CPU 216 transfers input informationincluding inserting coordinates data of an inserting point and insertioninclination angle data to the VR image creating apparatus 219 of theremote surgery supporting apparatus 201.

At a step S44, the VR image creating apparatus 219 of the remote surgerysupporting apparatus 201 receives inputs and inputs informationincluding coordinates data of an inserting point and insertioninclination angle data. Then, at a step S45, the CPU 257 of the VR imagecreating apparatus 219 determines a scale and/or direction of line ofview of a VR image based on the coordinates data of the inserting pointand insertion inclination angle data of the endoscope. At a step S46,the VR image constructing section 254 creates a VR image based on thescale and/or the direction of line of view and causes the VR imagedisplay monitor 255 to display the VR image through the display I/Fsection 256.

The VR image is displayed on a VR image display area 302 of a VR displayscreen 301, which is displayed on the VR image display monitor 255 asshown in FIG. 22. The VR display screen 301 includes a VR image displayarea 302, a two-dimensional image display area 303, an inserting pointdisplay field 304, a start/stop button 305 and a display scale changeinput portion 306. The VR image display area 302 displays a VR imagecreated by the VR image constructing section 254. The two-dimensionalimage display area 303 displays multiple two-dimensional CT imagesrelating to a VR image. The inserting point display field 304 displaysan inserting point (x0,y0,z0) of the rigid endoscope 202. The start/stopbutton 305 instructs the start and stop of tracking. The display scalechange input portion 306 is used for changing a display scale.

At a step S47, in the support information creating apparatus 220, theCPU 272 creates a support image 310 having the arrow image 309indicating the position of an affected part as shown in FIG. 23 on theendoscopic image 308 with reference to the VR image. The CPU 272 furthercauses the monitor 269 to display the support image 310 through thedisplay I/F section 270.

At the step S47, not only the support image 310 is created and/or isdisplayed but also support voice to an operation room is input throughthe microphone 267. The CPU 272 transmits the created support image dataand input support voice data to the surgery system 202 through thecommunications line 300.

In the surgery system 202 having received the support image data andinput support voice data, the support information creating apparatus 220displays the support image 310 on the monitor 232 and causes the speaker234 to play the support voice at a step S48.

Once the first support image 310 is displayed in this way and tracking(in accordance with live images of VR images) is started at a step S49,the CPU 216 of the system controller 204 measures insertion inclinationangle data of the rigid endoscope 203 by using the inclination anglesensor 222 at a step S50. At a step S51, the CPU 216 measures insertingamount data of the rigid endoscope 203 by using the amount-of-insertiondetecting section 221.

At a step S52, the CPU 216 judges whether or not either inserting amountdata or insertion inclination angle data is changed. If not changed, theprocessing returns to the step S50. If changed, the CPU 216 transfersinput information including inserting amount data and insertioninclination angle data to the VR image creating apparatus 219 of theremote surgery supporting apparatus 201 at a step S53.

When the VR image creating apparatus 219 of the remote surgerysupporting apparatus 201 receives (the input of) input informationincluding the inserting amount data and insertion inclination angle dataat a step S54, the CPU 257 of the VR image creating apparatus 219determines a scale and/or direction of line of view of a VR image basedon the inserting amount data and insertion inclination angle data at astep S55. At a step S56, the VR image constructing section 254 creates aVR image based on the scale and/or the direction of line of view andcauses the VR image display monitor 255 to display the VR image throughthe display I/F section 256.

Then, at a step S57, in the support information creating apparatus 220,the CPU 272 creates a support image 310 having an arrow image indicatingthe position of an affected part on endoscopic image data with referenceto the VR image and causes the monitor 269 to display the support image310 including (endoscopic image+instruction information) through thedisplay I/F section 270.

At the step S57, not only the support image 310 is created and/or isdisplayed but also support voice to an operation room is input throughthe microphone 267. The CPU 272 transmits the created support image dataand input support voice data to the surgery system 202 through thecommunications line 300.

In the surgery system 202 having received the support image data andinput support voice data, the support information creating apparatus 220displays the support image 310 on the monitor 232 and causes the speaker234 to play the support voice at a step S58.

Then, at a step S59, the CPU 216 of the system controller 204 judgeswhether or not an instruction for support termination from the remotecontroller 217 is given or not. If not, the processing returns to a stepS60. If so, the processing ends.

Through the processing at the steps S50 to S59, in the VR image creatingapparatus 219, when a live endoscopic image 214 a as shown in FIG. 24,for example, is displayed on the endoscopic image display monitor 214, ablood-vessel included virtual image 302 a without an organ part, forexample, as shown in FIG. 25 is displayed in the VR image display area302 on the VR display screen 301. Here, the blood-vessel includedvirtual image 302 a is in real time and has a same direction of line ofvision and size (scale) as those of the live endoscopic image 214 a.

When the rigid endoscope 2 is inclined from the state in FIG. 24 and thelive endoscopic image 214 b as shown in FIG. 26 is displayed on theendoscopic image display monitor 214, a blood-vessel included virtualimage 302 b without an organ part, for example, as shown in FIG. 27 isdisplayed in the VR image display area 302. Here, the blood-vesselincluded virtual image 302 b is in real time and has a same direction ofline of vision and size (scale) as those of the live endoscopic image214 b in accordance with (by tracking) the display.

In this way, according to this embodiment, by transmitting an insertingamount and insertion inclination angle of the rigid endoscope 203 to asupport room separate and far away from the operation room through thecommunications line 300, an instructing doctor in the remote supportroom provides the operator in the operation room with support images andsupport voice with reference to VR images tracking (in accordance with)live endoscopic images in real time. Thus, proper technique support canbe provided to the operator easily at low costs.

Fifth Embodiment

FIG. 28 is a configuration diagram showing configurations of a remotesurgery supporting apparatus and surgery system according to a fifthembodiment of the invention.

The fifth embodiment is substantially the same as the fourth embodiment.Therefore, only difference therebetween will be described, and the samereference numerals are given to the same components, the descriptions ofwhich will be omitted herein.

As shown in FIG. 28, the surgery system 202 includes a VR image displayapparatus 402 for receiving the input of VR image data created by the VRimage creating apparatus 219 of the remote surgery supporting apparatus201 and causes the VR image display monitor 401 to display the VR image.The VR image display apparatus 402 includes a video I/F section 403 anda display I/F section 404. The video I/F section 403 is used forinputting VR image data through information transfer I/F sections 224and 225 and the communications line 300. The display I/F section 404causes the VR image display monitor 401 to display a VR image based onthe input VR image data. The rest of the construction is the same asthat of the fourth embodiment.

According to this embodiment, at the step S47 or S57 according to thefourth embodiment, not only support image data and input support voicedata are transmitted to the surgery system 202 through thecommunications line 300 but also VR image data is transmitted to thesurgery system 202 through the communications line 300. The VR image isdisplayed on the VR image display monitor 401 in the operation room. Therest of the processing is the same as that of the fourth embodiment.

According to this embodiment, in addition to the advantages of thefourth embodiment, a support environment with a supporting doctor can bemore robust since an operator in an operation room can refer to a VRimage displayed on the VR image display monitor 401.

Sixth Embodiment

FIG. 29 is a configuration diagram showing configurations of a remotesurgery supporting apparatus and surgery system according to a sixthembodiment of the invention.

The sixth embodiment is substantially the same as the fifth embodiment.Therefore, only difference therebetween will be described, and the samereference numerals are given to the same components, the descriptions ofwhich will be omitted herein.

According to this embodiment, a VR image creating apparatus 501 isprovided in a surgery system. The VR image creating apparatus 501 hasthe similar configuration with that of a VR image creating apparatus 219in a remote surgery supporting apparatus 201 and creates a VR image tobe displayed on a VR image display monitor 555.

Like the VR image creating apparatus 219, the VR image creatingapparatus 501 includes a recording portion 551, a memory 552, acommunication I/F section 553, a VR image constructing section 554, adisplay I/F section 556 and a CPU 557. The recording portion 551 storesa CT image database (DB) including multiple CT images. The memory 552stores different kinds of programs. The communication I/F section 553receives inserting amount data detected by the amount-of-insertiondetecting section 221 and insertion inclination angle data detected bythe inclination angle sensor 222 from the system controller 204. The VRimage constructing section 554 constructs a VR image based on insertingamount data and insertion inclination angle data obtained by thecommunication I/F section 553 and a CT image in the CT image DB. Thedisplay I/F section 556 causes the VR image display monitor 555 todisplay a VR image constructed by the VR image constructing section 554.The CPU 557 controls these portions. A keyboard 558 and a mouse 559 usedfor inputting various kinds of data are connected to the CPU 557. Therest of the construction and operation is the same as that of the fifthembodiment.

According to this embodiment, in addition to the advantages of thefourth embodiment, a VR image is not required to transmit by the VRimage creating apparatus 219 of the remote surgery supporting apparatus201 through the communications line 300 since the VR image creatingapparatus 501 creates a VR image to be displayed on the VR image displaymonitor 555. Thus, the communication traffic of the communications line300 can be reduced, and the communication environment can be improvedsignificantly.

As described above, according to this embodiment, by providing a properinstruction from a remote facility with reference to a live endoscopicimage, surgery can be supported easily in real time at low costs.

Seventh Embodiment

A seventh embodiment of the invention will be described below withreference to drawings.

FIGS. 30 to 53 relate to the seventh embodiment of the invention. FIG.30 is a configuration diagram showing a configuration of a surgerysupporting apparatus. FIG. 31 is a diagram showing a state in which arigid endoscope in FIG. 30 is being used. FIG. 32 is a diagram showing aconstruction of the rigid endoscope in FIG. 31. FIG. 33 is a diagramshowing a construction of an essential part of a trocar in FIG. 31. FIG.34 is a top view showing a top face of an XY-inserting point measuringapparatus in FIG. 30. FIG. 35 is a side view showing a side of theXY-inserting point measuring apparatus in FIG. 30. FIG. 36 is a backview showing a back face of the XY-inserting point measuring apparatusin FIG. 30. FIG. 37 is a diagram showing a construction of a Z-insertingpoint measuring apparatus in FIG. 30. FIG. 38 is a flowchart showing aprocessing flow of a surgery support apparatus in FIG. 30. FIG. 39 is aflowchart showing a flow of XY-inserting point measuring processing inFIG. 38. FIG. 40 is a first diagram illustrating the XY-inserting pointmeasuring processing in FIG. 39. FIG. 41 is a second diagramillustrating the XY-inserting point measuring processing in FIG. 39.FIG. 42 is a third diagram illustrating the XY-inserting point measuringprocessing in FIG. 39. FIG. 43 is a flowchart showing a flow ofZ-inserting point measuring processing in FIG. 38. FIG. 44 is a firstdiagram illustrating the Z-inserting point measuring processing in FIG.43. FIG. 45 is a second diagram illustrating the Z-inserting pointmeasuring processing in FIG. 43. FIG. 46 is a diagram showing a VRdisplay screen displaying a VR image constructed/created by processingin FIG. 38. FIG. 47 is a diagram showing a first example of anendoscopic image displayed on an endoscopic image display monitor inFIG. 30. FIG. 48 is a diagram showing a VR display screen displayed inaccordance with the endoscopic image in FIG. 47. FIG. 49 is a diagramshowing a second example of an endoscopic image displayed on theendoscopic image display monitor in FIG. 30. FIG. 50 is a diagramshowing a VR display screen displayed in accordance with the endoscopicimage in FIG. 49. FIG. 51 is a diagram showing a third example of anendoscopic image displayed on the endoscopic image display monitor inFIG. 30. FIG. 52 is a diagram showing a VR display screen displayed inaccordance with the endoscopic image in FIG. 51. FIG. 53 is a diagramshowing a VR display screen displaying a VR image having a differentscale from that of the VR image in FIG. 52.

As shown in FIG. 30, a surgery supporting apparatus 601 according tothis embodiment has, in an operation room, a rigid endoscope 602, a VRimage creating apparatus 603, a system controller 604, a CCU 605, alight source apparatus 606, a pneumoperitoneum apparatus 607, anelectric knife 608, an ultrasonic treatment apparatus 609 and a VTR 610.

Image pickup signals picked up by an image pickup section 611 of therigid endoscope 602 are transmitted to the CCU 605 and undergo imageprocessing therein. Then, the result is output to the VTR 610 forrecording images and the system controller 604.

The system controller 604 includes a communication I/F section 612, amemory 613, a display I/F section 615 and a CPU 616. The communicationI/F section 612 exchanges setting information with the CCU 605, thelight source apparatus 606, the pneumoperitoneum apparatus 607, anelectric knife 608, an ultrasonic treatment apparatus 609, and the VTR610. The memory 613 stores different kinds of programs. The display I/Fsection 615 causes an endoscopic image display monitor 614 to displayimage signals from the CCU 605. The CPU 616 controls these portions. Aremote controller 617 is connected to the CPU 616 through thecommunication I/F section 612. Various kinds of data can be inputthrough the remote controller 617.

The rigid endoscope 602 includes an amount-of-insertion detectingsection 621, an inclination angle sensor 622, an XY-inserting pointmeasuring apparatus 625 and a Z-point inserting point measuringapparatus 627. The amount-of-insertion detecting section 621 detects aninserting amount of the rigid endoscope 602. The inclination anglesensor 622 detects an inclination angle of insertion of the rigidendoscope 602. The XY-inserting point measuring apparatus 625 has anoptical image sensor 623 and a switch 624. The optical image sensor 623measures XY-coordinates of an inserting point of the rigid endoscope602. The Z-point inserting point measuring apparatus 627 has anelectromagnetic sensor 626 for measuring a Z-coordinate of an insertingpoint of the rigid endoscope 602.

Based on a CT image obtained by a CT apparatus (not shown) in advance,the VR image creating apparatus 603 creates a volume rendering image (VRimage), which is a virtual image in real time and in a same direction ofline of vision as that of an endoscopic image picked up by the rigidendoscope 602.

More specifically, the VR image creating apparatus 603 includes a CTimage DB 631, a memory 632, a communication I/F section 633, a VR imageconstructing section 634, a display I/F section 636 and a CPU 637. TheCT image DB 631 is a recording portion for storing a CT image database(DB) including multiple CT images. The memory 632 stores different kindsof programs. The communication I/F section 633 exchanges data with thecommunication I/F section 612 for the amount-of-insertion detectingsection 621, the inclination angle sensor 622, the XY inserting pointmeasuring apparatus 625, the Z-point inserting point measuring apparatus627 and the system controller 604. The VR image constructing section 634constructs a VR image based on different kinds of data obtained by thecommunication I/F section 633 and a CT image in the CT image DB 631. Thedisplay I/F section 636 causes the VR image display monitor 635 todisplay a VR image constructed by the VR image constructing section 634.The CPU 637 controls these portions. A keyboard 638 and a mouse 639 usedfor inputting various kinds of data are connected to the CPU 637.

As shown in FIG. 31, the rigid endoscope 602 is inserted into the bodyof a patient 640 along with a treating apparatus 643 such as an electricknife and an ultrasonic treating apparatus through the trocars 641 and642.

As shown in FIG. 32, the rigid endoscope 602 includes an image pickupsection 611 at the inserted proximal end and the inclination anglesensor 622 at a handle 645 on the inserted proximal end side. Theinclination angle sensor 622 measures an insertion inclination angle ofthe rigid endoscope 602 by using a gyroscopic compass and outputs theresult to the VR image creating apparatus 603.

As shown in FIG. 33, the amount-of-insertion detecting section 621 isprovided on the proximal end side of the trocar 641 for guiding aninsert portion 646 of the rigid endoscope 602 into the body of thepatient 640. The amount-of-insertion detecting section 621 includes aroller 647 and a rotary encoder 648. The roller 647 is in contact withan outer surface of the insert portion 646 and rotates in accordancewith the insertion of the insert portion 646. The rotary encoder 648detects an amount of rotation of the roller 647 and outputs the amountof rotation to the VR image creating apparatus 603 as an amount ofinsertion of the insert portion 646.

As shown in FIGS. 34 to 36, the XY-inserting point measuring apparatus625 has substantially the same construction as that of a publicly knownoptical mouse. The XY-inserting point measuring apparatus 625 has theswitch 624 on the top face and a pointer 649 on the back face. Theswitch 624 is used for confirming an inserting point. The pointer 649 isused for printing a marker on the body surface of the patient 640 inconnection with the optical image sensor 623 and the switch 624. Theoptical image sensor 623 detects a moved amount.

As shown in FIG. 37, the Z-point inserting point measuring apparatus 627includes a fixing member 652, a trocar holding portion 653 and anmeasuring portion 654. The fixing member 652 supports and fixes asupport rod 651 in perpendicular to an operation table 650 on which thepatient 640 lies. The trocar holding portion 653 holds a trocar 641extending from the support rod 651 at the right angle. The measuringportion 654 self-contains an electromagnetic induction sensor 626 whichdetects a moved amount in the axial direction (Z-direction) of thesupport rod 651 of the trocar holding portion 653 and outputs the resultto the VR image creating apparatus 603.

Operations of this embodiment having the above-described constructionwill be described. As shown in FIG. 38, XY-inserting point measuringprocessing by the XY-inserting point measuring apparatus 625 isperformed at a step S61. Details of the XY-inserting point measuringprocessing will be described later.

After an XY-inserting point is measured by the XY-inserting pointmeasuring apparatus 625, the trocar 641 is placed at a position markedby a pointer 649 of the XY-inserting point measuring apparatus 625 andis inserted into the body of the patient 640 at a step S62. At a stepS63, Z-point measuring processing is performed by the Z-inserting pointmeasuring apparatus 27. Details of the Z-inserting point measuringprocessing will be described later.

In the processing at the steps S61 to S63, an inserting point of therigid endoscope 602 is determined. The rigid endoscope 602 is insertedthrough the trocar 641. Then, at a step S64, an insertion inclinationangle, that is, an attitude angle of the rigid endoscope 602 is measuredby the inclination angle sensor 622. At a step S65, the direction ofline of vision of an endoscopic image to be picked up by the rigidendoscope 2 is determined based on the insertion inclination angle at astep S65.

Once the insertion of the rigid endoscope 602 is started at a step S66,an amount of insertion of the rigid endoscope 602 is measured by theamount-of-insertion detecting section 621 at a step S67. At a step S68,a display scale of a VR image is determined based on the insertingamount (in accordance with the distance, that is, higher scale near anorgan while lower scale away from the organ).

Once a direction of the line of vision and a display scale aredetermined, a VR image is created by the VR image constructing section634 based on the direction of the line of vision and the display scaleat a step S69. At a step S70, the VR image is displayed on the VR imagedisplay monitor 635 through the display I/F section 636, and theprocessing ends.

In the XY-inserting point measuring processing at the step S61, astarting point is set at “the navel” of the patient 640 at a step S71 asshown in FIG. 39. The XY-inserting point measuring apparatus 625 isplaced on “the navel” as shown in FIG. 40. At a step S72, the switch 624is pressed down. Thus, the origin (0,0) can be determined on the xyplane.

Next, at a step S73, the XY inserting point measuring apparatus 625 ismoved to a position of the insertion of the trocar 641 as shown in FIG.41. By pressing down the switch 624 at a step S74, a marker is stampedat the position as shown in FIG. 42 at a step S75. Thus, an insertingpoint (x0,y0) can be determined on the XY plane of the rigid endoscope602.

In the Z-inserting point measuring processing at the step S63, after thetrocar 641 held at the Z-point inserting point measuring apparatus 627is placed and inserted at the inserting point (x0,y0), the position ofthe origin (x0,y0,0) in the z direction is detected, and measuring aZ-inserting point is started as shown in FIG. 44 upon starting the gassupply of the pneumoperitoneum apparatus 7 at step S81.

At a step S82, pressure of the abdominal cavity is set at a set pressureby the pneumoperitoneum apparatus 607. At a step S83, a moved amount ΔZof the trocar 641 at the set pressure is measured by the Z-pointinserting point measuring apparatus 627 as shown in FIG. 45. At a stepS84, the moved amount ΔZ is an inserting point z0 in the Z direction ofthe rigid endoscope 602.

By performing the XY inserting point measuring processing and theZ-inserting point measuring processing, an inserting point (x0,y0,z0) ofthe rigid endoscope 602 is determined.

Next, a VR display screen to be displayed on the VR image displaymonitor 635 will be described. As shown in FIG. 46, a VR display screen701 includes a VR image display area 702, a two-dimensional imagedisplay area 703 and an inserting point display field 704 and so on. TheVR image display area 702 displays a VR image created by the VR imageconstructing section 634. The two-dimensional image display area 703displays multiple two-dimensional CT images relating to a VR image. Theinserting point display field 704 displays inserting point (x0,y0,z0)(=(@@,@@,@@)) of the rigid endoscope 602 determined by the XY-insertingpoint measuring processing and Z-inserting point measuring processing.

For example, when a live endoscopic image 614 a as shown in FIG. 47 isdisplayed on the endoscopic image display monitor 614, a blood-vesselincluded virtual image 702 a without an organ part, for example, asshown in FIG. 48 is displayed in the VR image display area 702 on the VRdisplay screen 701. Here, the blood-vessel included virtual image 702 ais in real time and has a same direction of line of vision and size(scale) as those of the live endoscopic image 614 a.

When the rigid endoscope 2 is inclined from the state in FIG. 47 and thelive endoscopic image 614 b as shown in FIG. 49 is displayed on theendoscopic image display monitor 614, a blood-vessel included virtualimage 702 b without an organ part, for example, as shown in FIG. 50 isdisplayed in the VR image display area 702. Here, the blood-vesselincluded virtual image 702 b is in real time and has a same direction ofline of vision and size (scale) as those of the live endoscopic image614 b in accordance with (by tracking) the display.

When the live endoscopic image 614 c as shown in FIG. 51 is displayed onthe endoscopic image display monitor 614, a blood-vessel includedvirtual image 702 c without an organ part, for example, as shown in FIG.52 is displayed in the VR image display area 702 as described above.Here, the blood-vessel included virtual image 702 c is in real time andhas a same direction of line of vision and size (scale) as those of thelive endoscopic image 614 c. In this case, by manipulating the keyboard638, for example, a blood vessel included virtual image 702 d having adifferent scale as shown in FIG. 53 can be displayed in the VR imagedisplay area 702. The scale may be set arbitrarily. FIG. 53 shows astate magnified by 1.5 times, and the magnification scale is displayedin a scale display area 710.

In this way, according to this embodiment, an inserting point, insertioninclination angle, inserting amount of the rigid endoscope 602 aremeasured, and a real time VR image having a same direction of line ofvision and size (scale) as those of a live endoscopic images is createdand displayed based on the data of the inserting point, insertioninclination angle and inserting amount. Thus, information required forimplementing a technique (such as blood-vessel included information) canbe visually checked, and the technique can be supported safely andproperly.

As described above, according to this embodiment, surgery can beadvantageously supported by providing virtual images corresponding tolive endoscopic images easily and in real time.

Eighth Embodiment

FIGS. 54 to 71 relate to an eighth embodiment of the invention. FIG. 54is a construction diagram showing a construction of a technique supportsystem. FIG. 55 is a block diagram showing an essential configuration ofthe technique support system in FIG. 54. FIG. 56 is a diagram showing aconstruction of an endoscope in FIG. 54. FIG. 57 is a diagramillustrating an operation of the technique support system in FIG. 54.FIG. 58 is a flowchart showing a processing flow of the techniquesupport system in FIG. 54. FIG. 59 is a first diagram showing a screendeveloped in the processing in FIG. 58. FIG. 60 is a second diagramshowing a screen developed in the processing in FIG. 58. FIG. 61 is athird diagram showing a screen developed in the processing in FIG. 58.FIG. 62 is a first diagram illustrating a variation example of anoperation of the technique support system in FIG. 54. FIG. 63 is asecond diagram illustrating a variation example of an operation of thetechnique support system in FIG. 54. FIG. 64 is a diagram showing afirst variation example of a screen developed in the processing in FIG.58. FIG. 65 is a first diagram showing a second variation example of ascreen developed in the processing in FIG. 58. FIG. 66 is a seconddiagram showing a second variation example of a screen developed in theprocessing in FIG. 58. FIG. 67 is a third diagram showing the secondvariation example of a screen developed in the processing in FIG. 58.FIG. 68 is a diagram illustrating a side-view observation direction of aside-view endoscope in FIG. 54. FIG. 69 is a flowchart showingprocessing for correcting a general virtual image, which is compliantwith the side-view endoscope in FIG. 68. FIG. 70 is a first diagramshowing a screen developed by the processing in FIG. 69. FIG. 71 is asecond diagram showing a screen developed by the processing in FIG. 69.

As shown in FIG. 54, a technique support system 801 according to thisembodiment is combined with an endoscope system. More specifically, thetechnique support system 801 has an endoscope 802 as observation means,which can observe the inside of a body cavity of a body to be examined,a CCU 804, a light source 805, an electric knife apparatus 806, apneumoperitoneum apparatus 807, an ultrasonic drive power supply 808, aVTR 809, a system controller 810, a virtual image creating section 811,a remote controller 812A, a voice input microphone 812B, a mouse 815, akeyboard 816, a virtual image display monitor 817 and an endoscopicimage monitor 813 and virtual image monitor 817 a in an operation room.

As the endoscope 802 according to this embodiment, a laparoscope is usedas shown in FIG. 56. The endoscope (laparoscope) 802 has an insertportion 802 b to be inserted into an abdominal cavity of a body to beexamined and a handle 802 a disposed on the proximal end side of theinsert portion 802 b. An illumination optical system and an observationoptical system are provided within the insert portion 802 b. Theillumination optical system and the observation optical systemilluminate a part to be observed within an abdominal cavity of a body tobe examined, and an observation image of the inside of the abdominalcavity of the body to be examined can be obtained.

A light guide connector 802 c is provided at the handle 802 a. One endof a light guide cable 802 f (refer to FIG. 54) is connected to thelight guide connector 802 c having the other end connecting to the lightsource apparatus 805. Thus, illumination light from the light sourceapparatus 805 can be irradiated to a part to be observed through theillumination optical system within the insert portion 802 b.

A camera head 802 d having image pickup means such as a CCD is connectedto an eyepiece (not shown) provided at the handle 802 a. A remote switch802 g to be used for performing an operation such as zooming in/out ofan observation image is provided in the camera head 802 d. A cameracable 802 e is extended from the proximal end side of the camera head802 d. A connection connector (not shown) for electrically connecting tothe CCU 804 is provided at the other end of the camera cable 802 e.

Referring back to FIG. 54, during surgery, the endoscope 802 is providedwithin a trocar 837 and is held at the abdominal part within the body ofa patient by the trocar 837. By keeping this state, the insert portionof the endoscope 802 is inserted into the abdomen area, and the abdomenarea is picked up by the image pickup section such as a CCD. Then, thepicked-up image pickup signals are supplied to the CCU 804 through thecamera head 802 d.

The CCU 804 performs signal processing on the image pickup signals fromthe endoscope 802 and supplies image data (such as endoscopic live imagedata) based on the image pickup signals to the system controller 810 inan operation room. Under the control of the system controller 810, imagedata based on a still image or moving images of endoscopic live imagesis selectively output from the CCU 804 to the VTR 809. A detailconstruction of the system controller 810 will be described later.

Under the control of the system controller 810, the VTR 809 can recordor play endoscopic live image data from the CCU 804. During the play,the played endoscopic live image data is output to the system controller810.

The light source apparatus 805 is a light source apparatus for supplyingillumination light to the endoscope 2 through a light guide.

The electric knife apparatus 806 is a surgical treatment apparatus forcutting an abnormal part within the abdomen area of a patient, forexample, by using electric heat of an electric knife probe. Theultrasonic drive power supply 808 is a surgical treatment apparatus forcutting or coagulating the abnormal part by using an ultrasonic probe(not shown).

The pneumoperitoneum apparatus 807 has air supply and air-intake units,not shown. The pneumoperitoneum apparatus 807 supplies carbon dioxide tothe abdomen area, for example, within the body of a patient through thetrocar 837 connecting to the pneumoperitoneum apparatus 807.

The light source apparatus 805, the electric knife apparatus 806, thepneumoperitoneum apparatus 807 and the ultrasonic drive power supply 808are electrically connected to the system controller 810 and are drivenunder the control of the system controller 810.

In addition to various kinds of equipment including the CCU 804, the VTR809, the light source apparatus 805, the electric knife apparatus 806,the pneumoperitoneum apparatus 807 and the ultrasonic drive power supply808, the system controller 810, the endoscopic image monitor 813, and avirtual image monitor 817 a are placed within an operation room.

According to this embodiment, in order to perform treatment at aposition as shown in FIG. 54 on the patient 830 by an operator 831 whopicks up images of a body to be examined by inserting the insert portioninto the abdominal part of the patient 830 through the trocar 837, theendoscopic image monitor 813 and the virtual image monitor 817 a areplaced at an easy-to-see position (in the direction of the field ofvision) with respect to the position of the operator 831.

The system controller 810 controls different kinds of operations (suchas display control and dimming control) of the entire endoscope system.As shown in FIG. 55, the system controller 810 has a communicationinterface (called communication I/F, hereinafter) 818, a memory 819, aCPU 820 as a control portion and a display interface (called displayI/F, hereinafter) 821.

The communication I/F 818 is electrically connected to the CCU 804, thelight source apparatus 805, the electric knife apparatus 806, thepneumoperitoneum apparatus 807, the ultrasonic drive power supply 808,the VTR 809 and the virtual image creating section 811, which will bedescribed later. The exchange of drive control signal therefor and theexchange of endoscopic image data are controlled by the CPU 820. Aremote controller 812A and voice input microphone 812B for an operatoras remote operation means are electrically connected to thecommunication I/F 818. The communication I/F 818 captures operationinstruction signals from the remote controller 812A and voiceinstruction signals from the voice input microphone 812B and suppliesthese signals to the CPU 820.

Though not shown, the remote controller 812A has a white balance button,a pneumoperitoneum button, a pressure button, a record button, a freezebutton, a release button, a display button, an operation button forimplementing two-dimensional display (2D display) for creating virtualimages, an operation button for implementing three-dimensional display(3D display) for displaying virtual images, an inserting point button, afocus point button, buttons for instructing to change a display scalefor 3D display (such as a zoom-in button and a zoom-out button), adisplay color button, a tracking button, an operation button forswitching and/or determining setting input information for an operationsetting mode determined by pressing one of buttons, a numeric keypad.The white balance button is used for display images displayed on anendoscopic image monitor 813 for endoscopic live images, the virtualimage display monitor 817 or the virtual image monitor 817 a. Thepneumoperitoneum button is used for implementing the pneumoperitoneumapparatus 807. The pressure button is used for increasing or decreasingthe pressure for implementing a pneumoperitoneum. The record button isused for recording endoscopic live images in the VTR 809. The freezebutton and the release button are used for recording. The display buttonis used for displaying endoscopic live images or virtual images. Theoperation button for 2D display may includes an axial button, coronalbutton, and sagittal button in accordance with one of different kinds of2D display mode. The inserting point button is used for indicating adirection of field of view of a virtual image displayed in a 3D displaymode (and may be a button for displaying information on insertion to anabdomen area of the endoscope 802 such as numerical values in X-, Y- andZ-directions of the abdomen area to which the endoscope 802 isinserted). The focus button is a button for displaying a numerical valueof the axial direction (angle) of the endoscope 802 inserted into theabdomen area). The display color button is used for change a displaycolor. The tracking button is used for tracking. The numeric keypad isused for inputting numeric values and so on.

Thus, by using the remote controller 812A (or switch) including thesebuttons, an operator can operate to obtain desired information fast.

The memory 819 stores image data of endoscopic still images and datasuch as equipment setting information, for example. The data storing andreading are controlled by the CPU 820.

The display I/F 821 is electrically connected to the CCU 804, the VTR809 and the endoscopic image monitor 813. The display I/F 821 exchangesendoscopic live image data from the CCU 4 or endoscopic image datahaving been played by the VTR 809 and outputs the received endoscopiclive image data to the endoscopic image monitor 813. Thus, theendoscopic image monitor 813 displays endoscopic live images based onthe supplied endoscopic live image data.

The endoscopic image monitor 813 can display not only endoscopic liveimages but also display setting information such as setting states andparameters of the apparatuses of the endoscope system under the displaycontrol of the CPU 820.

The CPU 820 controls different kinds of operations in the systemcontroller 810, that is, performs control over exchanges of differentkinds of signals by the communication I/F 818 and the display I/F 824,control over writing and/or reading of image data to/from the memory819, control over display by the endoscopic image monitor 813, andcontrol over different kinds of operations based on operation signalsfrom the remote controller 812A (or switch).

On the other hand, the system controller 810 is electrically connectedto the virtual image creating section 811.

As shown in FIG. 55, the virtual image creating section 811 has a CTimage DB section 823, a memory 824, a CPU 825, a communication I/F 826,a display I/F 827 and switching section 827A.

The CT image DB section 823 includes a CT image data capturing portion(not shown) for capturing three-dimensional image data created by apublicly known CT apparatus, not shown, for imaging an X-ray tomographicimage of a patient through a portable memory medium such as amagneto-optical (MO) disk and a digital versatile disk (DVD). Thus, theCT image DB section 823 can store the captured three-dimensional imagedata (CT image data). The reading and writing of the three-dimensionalimage data are controlled by the CPU 825.

The memory 824 stores the three-dimensional image data and data such asvirtual image data created by the CPU 825 based on the three-dimensionalimage data. Thus, the storing and reading of these kinds of data arecontrolled by the CPU 825.

The communication I/F 826 is connected to the communication I/F 818 ofthe system controller 810 and exchanges control signals required forperforming different kinds of operations in connection with the virtualimage creating section 811 and the system controller 810. Thecommunication I/F 826 is controlled by the CPU 825, and the controlsignals are captured into the CPU 825.

The display I/F 827 outputs virtual images created under the control ofthe CPU 825 to the virtual image monitors 817 and 817 a through theswitching section 827A. Thus, the virtual image monitors 817 and 817 adisplay supplied virtual images. In this case, under the switchingcontrol of the CPU 825, the switching section 827A can switch the outputof the virtual images and output the virtual images to the selected oneof the virtual image monitors 817 and 817 a. When switching the displayof virtual images is not required, the switching section 827A is notrequired. A same virtual image may be displayed on both of the virtualimage monitors 817 and 817 a.

The mouse 815 and the keyboard 816 are electrically connected to the CPU825. The mouse 815 and the keyboard 816 are operation means to be usedfor inputting and/or setting different kinds of setting informationrequired for performing an operation for displaying virtual images bythe virtual image display apparatus.

The CPU 825 performs different kinds of operations in the virtual imagecreating section 811, that is, performs control over exchanges differentkinds of signals by the communication I/F 826 and the display I/F 827,control over writing and/or reading of image data to/from the memory824, control over display by the monitors 817 and 817 a, control overswitching of the switching section 827A, and control over differentkinds of operations based on operation signals from the mouse 815 and/orthe keyboard 816.

According to this embodiment, the virtual image creating section 811 maybe connected to a remotely provided virtual image creating section, forexample, through communication means so as to be constructed as a remotesurgery support system.

According to this embodiment, as shown in FIG. 56, the sensor 803 isprovided at the handle 802 a of the endoscope 802 in order to create anddisplay virtual images based on a direction of field of vision of theendoscope 802. The sensor 803 accommodates a gyroscopic sensor, forexample, and detects information such as an angle of insertion into theabdomen area of the endoscope 802. The detection information of thesensor 803 is supplied to the virtual image creating section 811 throughthe communication I/F 826 as shown in FIG. 55.

While the sensor 803 is electrically connected to the virtual imagecreating section 811 through a cable according to this embodiment, thesensor 803 may be connected to the virtual image creating section 811 ina wireless manner so as to implement data communication.

Next, operations of this embodiment having the above-describedembodiment will be described. According to this embodiment, based onangle information of insertion of the endoscope 802 into the abdomenarea by the sensor 803, the virtual image creating section 811 creates avirtual image in the normal direction (front) with respect to a part ofconcern (abnormal part 901 near a target organ 900 as shown in FIG. 57,which corresponds to the field of vision of the endoscope 802. At thesame time, the virtual image creating section 811 creates multiple sidevirtual images viewed from the right, left, upper, lower and back of acube with respect to the part of concern (abnormal part) 901 near thetarget organ 900. While the virtual image creating section 811 createsimages of the right, left, upper, lower and back sides of the cube withrespect to the part of concern abnormal part) 901 as planes havingpredetermined angles about an observation image plane in the directionof the field of vision of the endoscope 802, other planes of the sidesin different directions may be adopted.

A virtual image at least in the normal direction (front plane) iscreated in real time in synchronization with live endoscopic images ofthe endoscope 802 based on detection information of the sensor 803.

According to this embodiment, multiple virtual images of the right,left, upper, lower back planes may be created in real time insynchronization with live endoscopic images like virtual images in thenormal direction (front view). However, according to this embodiment, asdescribed later, a side virtual images is created as a still image basedon a frame image of the virtual image in the normal direction (frontview) when an instruction for displaying the side virtual image isgiven.

Once a technique is started and an observation image of the inside of abody to be examined is picked up by the camera head 802 d, an endoscopicimage is displayed on the endoscopic image monitor 813.

Then, as shown in FIG. 58, the virtual image creating section 811creates a virtual image in the normal direction (front view) based onangle information of insertion of the endoscope 802 into the abdomenarea by the sensor 803 at a step S91. Then, a normal-direction virtualscreen 950 is displayed on the virtual image monitor 817 a as shown inFIG. 59.

The normal-direction virtual screen 950 in FIG. 59 includes commandbuttons 1002, 1003, 1004, 1005 and 1006 for instructing to displaymultiple side virtual images viewed from the right, left, upper, lowerand back directions in addition to a normal-direction (front view)virtual image 1001, and a multi-command button 1007 for instructing todisplay the sides at the same time.

Then, at a step S92, one of the command buttons 1002, 1003, 1004, 1005and 1006 and the multi-command button 1007 is selected by a pointer 1000on the normal-direction virtual screen 950. Then, it is judged whetheror not a display with another point of vision is implemented.

While the selection by the pointer 1000 is performed by using a pointingdevice or the like above, the operator may select by voice by using thevoice input microphone 812B. For example, by producing a sound, “BACK”,the back view may be selected by voice recognition.

When one of the command buttons 1002, 1003, 1004, 1005 and 1006 and themulti-command button 1007 is selected by the pointer 1000 on thenormal-direction virtual screen 950, a virtual image from a point ofvision corresponding to a command button selected at the step S93 isdisplayed on the virtual image monitor 817 a.

For example, when the command button 1002 for displaying right-sidedisplay as shown in FIG. 59 is selected by the pointer 1000, a differentpoint-of-vision virtual screen 951 having the right-side virtual image1011 instead of the virtual image 1001 in the normal direction (frontview) as shown in FIG. 60 is displayed on the virtual image monitor 817a.

The different point-of-vision virtual screen 951 in FIG. 60 includes theright-side virtual image 1011, the command buttons 1003, 1004, 1005 and1006 for instructing to display multiple side virtual images of theleft, upper, lower and back views, the multi-command button 1007 forinstructing to display sides at the same time, and a normal displaybutton 1008 for instructing to display a normal-direction (front view)virtual image 1001.

At a step S94, an internal timer within the CPU 825 of the virtual imagecreating section 811 is set, and measuring a time is started.

Subsequently, at a step S95, it is judged whether the normal displaybutton 1008 is selected by the pointer 1000 on the differentpoint-of-vision virtual screen 951. If the normal display button 1008 isselected, the processing returns to the step S91. If the normal displaybutton 1008 is not selected, it is judged whether or not one of thecommand buttons 1003, 1004, 1005 and 1006 and the multi-command button1007 is selected by the pointer 1000 on the different point-of-visionvirtual screen 951 at a step S96.

If one of the command buttons 1003, 1004, 1005 and 1006 and themulti-command button 1007 is selected by the pointer 1000 on thedifferent point-of-vision virtual screen 951, a virtual image at a pointof vision in accordance with a command button selected is displayed onthe virtual image monitor 817 a at a step S97. At a step S98, theinternal timer within the CPU 825 is reset, and a time measurement isrestarted. Then, the processing goes to a step S99. If a command buttonis not selected, the processing goes from the step S96 to the step S99directly.

At the step S99, the CPU 825 judges whether or not live endoscopicimages of the endoscope 802 has a predetermined amount of movement basedon a motion vector due to image processing by the CPU 820 of the systemcontroller 810. If the live endoscopic images have a predeterminedamount of movement or larger than the predetermined amount, theprocessing returns to the step S91. If the live endoscopic images doesnot have the predetermined amount of movement or larger than thepredetermined amount, it is judged whether or not a predetermined amountof time has passed in the internal timer within the CPU 825 at a stepS100. If the predetermined amount of time has passed, the processingreturns to the step S91. If the predetermined amount of time has notpassed, the processing returns to the step S95.

At the step S93 or step S97, if the multi-command button 1007 isselected by the pointer 1000, a multi-point-of-vision virtual screen 952as shown in FIG. 61 is displayed on the virtual image monitor 817 a. Themulti-point-of-vision virtual screen 952 has images including the normaldirection (front view) virtual image and multiple side virtual images ofthe right, left, upper, lower, back views with the normal direction(front view) virtual image as the center.

As described above, according to this embodiment, biological imageinformation of different views of the surroundings of a part of concern(abnormal part) (such as image information having arteries and veins,which are hidden by organs and image information of the position of thepart of concern) can be provided to an operator for a predeterminedperiod of time during a technique. If live endoscopic images have apredetermined amount of movement or larger than the predetermined amount(change), the normal direction (front view) virtual image based on theangle information of insertion of the endoscope 2 to the abdomen areacan be displayed again. Thus, various virtual images can be providedduring the technique in real time.

According to this embodiment, while, as shown in FIG. 62, a side virtualimage orthogonal to the normal-direction (front view) virtual image iscreated, the invention is not limited thereto. As shown in FIG. 63, aside virtual image of a side displaced by an offset angle θ from theplane orthogonal to the normal direction (front view) virtual image maybe created. The plane of the side virtual image according to thisembodiment is just an example, and the advantages of this embodiment canbe obtained as far as the plane of the side virtual image is anarbitrary plane suitable for a technique.

While, according to this embodiment, an image to be displayed on thevirtual image monitor 817 a is one of the normal direction (front view)virtual image and the side virtual image, the invention is not limitedthereto. For example, as shown in FIG. 64, a (right) side virtual imagemay be displayed next to the normal direction (front view) virtualimage.

Instead of the command buttons, as shown in FIG. 65, thumbnail images ofthe side virtual images may be displayed. As shown in FIG. 66, byhighlighting the frame of a thumbnail image of a selected side virtualimage, the displayed side virtual image can be identified easily.Furthermore, a state of another side virtual image can be visuallyrecognized through the thumbnail image display. Thus, the side virtualimage can be provided more effectively. FIG. 67 shows a display exampleof a thumbnail image of a side virtual image, in which a (right) sidevirtual image is displayed next to the normal-direction (front view)virtual image.

By the way, according to this embodiment, the endoscope 802 is astraight vision endoscope rather than a diagonal vision endoscope. Thus,as shown in FIG. 68, since the direction of insertion of the diagonalvision endoscope 990 does not agree with the side vision direction, animage is obtained which has a different direction of the direction ofthe normal-direction (front view) only depending on the live endoscopicimages for diagonal vision and the direction of insertion of theendoscope.

Accordingly, the CPU 825 of the virtual image creating section 811corrects the normal-direction (front view) virtual image of the diagonalvision endoscope 990 and determines an observation direction as follows.

As shown in FIG. 69, at a step S101, a straight vision virtual image 991in the direction of insertion of the diagonal vision endoscope 990 asshown in FIG. 70 is displayed on the virtual image monitor 817 a. At astep S102, a correction angle corresponding to the diagonal vision angleof diagonal vision endoscope 990 on the virtual image 991 is input, anda diagonal vision correction button is selected. Thus, at a step S103, adiagonal vision corrected virtual image 992 as shown in FIG. 71 isdisplayed on the virtual image monitor 817 a. When a straight-visiondirection display button is selected at a step S104 while the diagonalvision corrected virtual image 992 is being displayed, the processingreturns to the step S101. If the straight-vision direction displaybutton is not selected, the processing returns to the step S102.

When the OK button is selected while the straight-vision virtual image991 or the diagonal vision corrected virtual image 992 is beingdisplayed, the straight-vision virtual image 991 or the diagonal visioncorrected virtual image 992 is registered as a normal-direction (frontview) virtual image.

Thus, normal direction (front view) virtual images compliant with thestraight-vision type and the diagonal vision type can be obtained.

A normal-direction (front view) virtual image having a same direction asthe direction of an observation image of a side-vision endoscope can beobtained from a side-vision endoscope as well as from a diagonal visionendoscope by performing a same angle correction (90 degree correction)as the angle correction for a diagonal vision endoscope.

As described above, according to this embodiment, a virtual imagesuitable for technique support can be advantageously provided during atechnique in real time.

Ninth Embodiment

FIGS. 72 to 82 relate to a ninth embodiment. FIG. 72 is a constructiondiagram showing a construction of a technique support system. FIG. 73 isa block diagram showing an essential configuration of the techniquesupport system in FIG. 72. FIG. 74 is a diagram showing a constructionof an endoscope in FIG. 72. FIG. 75 is a diagram illustrating anoperation of the technique support system in FIG. 72. FIG. 76 is aflowchart showing a processing flow of the technique support system inFIG. 72. FIG. 77 is a first diagram showing a screen developed by theprocessing in FIG. 76. FIG. 78 is a second diagram showing a screendeveloped by the processing in FIG. 76. FIG. 79 is a first diagramillustrating an operation in which a point-of-vision information inputportion in FIG. 72 is a sensor provided at a handle of the endoscope.FIG. 80 is a second diagram illustrating an operation in which thepoint-of-vision information input portion in FIG. 72 is a sensorprovided at the handle of the endoscope. FIG. 81 is a diagram showing ahead band having the point-of-view input portion in FIG. 72. FIG. 82 isa diagram showing a state that the head band in FIG. 81 is worn.

In the following description of the ninth embodiment, the same referencenumerals are given to the same components as those of the eighthembodiment, the descriptions of which will be omitted.

As shown in FIG. 72, a technique support system 801 according to thisembodiment further includes a point-of-view information input portion1103 having a joystick or the like. The point-of-vision informationinput portion 1103 is connected to a communication I/F 826 as shown inFIG. 73.

Next, operations of this embodiment having the above-describedembodiment will be described. According to this embodiment, based onangle information of insertion of the endoscope 802 into the abdomenarea by the sensor 803, the virtual image creating section 811 creates avirtual image in a direction with respect to a part of concern (abnormalpart 901 near a target organ 900 as shown in FIG. 75, which correspondsto the field of vision of the endoscope 802. More specifically, as shownin FIG. 75, serial virtual images resulting from a process in which aposition of the point of vision toward the part of concern 901 movesfrom a position of an inserting point of the endoscope to a virtualpoint of vision specified by a point-of-vision position changing portionsuch as a mouse, a joystick and a footswitch. As shown in FIG. 75, whena point of vision is moved arbitrarily on a sphere of the movement ofthe point of vision by the point-of-vision position changing portion,virtual images are created in series in accordance with the movement ofthe point of vision. In other words, serial virtual images are createdbased on an axial angle with respect to the part of concern, that is, anaxial angle specifying a position of the point of vision or an axialangle or position of point of vision specified by a point-of-visionposition specifying portion.

At least a virtual image is created in real time in synchronization withlive endoscopic images of the endoscope 802 based on detectioninformation of the sensor 803.

Once a technique is started and the inside of a body to be examined isimaged by the camera head 802, an endoscopic image is displayed on theendoscopic image monitor 813.

Then, as shown in FIG. 76, the virtual image creating section 811creates a virtual image in the normal direction based on angleinformation of insertion of the endoscope 802 into the abdomen area bythe sensor 803 at a step S111. Then, a normal-direction virtual screen950 is displayed on the virtual image monitor 817 a as shown in FIG. 77.

The normal-direction virtual screen 950 in FIG. 77 includes a panoramacommand button 1151 for instructing panorama display, which is a displayof virtual images in accordance with the movement of the point of visionin addition to a normal-direction virtual image 1001.

Then, at a step S112, it is judged whether or not the panorama commandbutton 1151 is selected by a pointer 1152 on the normal-directionvirtual screen 950.

While the selection by the pointer 1152 is performed by using a pointingdevice above, the operator may select by voice by using the voice inputmicrophone 812B, for example. (For example, by producing a sound,“BACK”, the back view may be selected by voice recognition.)

When the panorama command button 1152 is selected by the pointer 1152 onthe normal-direction virtual screen 950 in FIG. 77, the panorama virtualscreen 961 having the panorama virtual image 1111 as shown in FIG. 78 isdisplayed on the virtual image monitor 817 a at a step S113.

The panorama virtual screen 961 in FIG. 78 includes a normal displaybutton 1008 and a virtual point-of-vision navigator 1009. The normaldisplay button 1008 is used for instructing to display anormal-direction virtual image 1001. The virtual point-of-visionnavigator 1009 indicates a relative position of a virtualpoint-of-vision with respect to the endoscope 802 of the panoramavirtual image 1111.

At a step S114, it is judged whether or not the normal display button1008 is selected.

Thus, according to this embodiment, a normal-direction virtual image anda panorama virtual image can be provided during surgery, and biologicalimage information of different views of the surroundings of a part ofconcern (abnormal part) (such as image information having arteries andveins, which are hidden by organs and image information of the positionof the part of concern) can be provided. Therefore, virtual imagessuitable for technique support can be provided during a technique inreal time.

While the point-of-vision input portion 1103 is a joystick, the sensor803 at the handle 802 a of the endoscope 802 may be the point-of-visioninformation input portion 1103. When the sensor 803 is thepoint-of-vision information input portion 1103, a panorama virtual imagefor the point of vision moved by a predetermined angle θ resulting fromthe inclination of the endoscope 802 by the angle θ as shown in FIGS. 79and 80. FIG. 79 shows an example that a panorama virtual image is movedin the opposite direction of that of the rotation angle of the endoscope802 while FIG. 80 shows an example that a panorama virtual image ismoved in the same direction as that of the rotation angle of theendoscope 802.

As shown in FIG. 81, a motion sensor 1154 as the point-of-visioninformation input portion 1103 is provided in a one-touch head band1153, which can be sterilized. The head band 1153 may be attached to thehead of an operator as shown in FIG. 82.

According to this embodiment, a virtual image suitable for techniquesupport can be provided in real time during a technique.

Tenth Embodiment

FIGS. 83 to 91 show a virtual image display apparatus according to atenth embodiment of the invention. FIG. 83 is a schematic constructiondiagram showing an entire construction of an endoscope system includingthe virtual image display apparatus. FIG. 84 is a block diagram showingan entire configuration of the endoscope system in FIG. 83. FIG. 85 is aperspective view showing an external construction of the endoscope inFIG. 83. FIG. 86 is a perspective view showing a construction example inwhich the system is attached to the arm of an operator. FIG. 87 is aperspective view showing an external construction of a trocar, which isan attachment target portion to which a sensor is attached. FIG. 88 is aconstruction perspective view showing a first variation example of theattachment target portion. FIG. 89 is a construction perspective viewshowing a second variation example of the attachment target portion.FIGS. 90 and 91 are diagrams illustrating a display operation of thisembodiment. FIG. 90 shows a display example of an operator monitor shownin FIG. 83. FIG. 91 is a flowchart illustrating main control processingby a CPU of a virtual image creating section.

The same reference numerals are given to the same components as those ofthe eighth embodiment, the descriptions of which will be omitted.

As shown in FIG. 83, a virtual image display apparatus 1201 according tothis embodiment is combined with an endoscope system. More specifically,the virtual image display apparatus 1201 has an endoscope 802 asobservation means, a sensor 1203 a, an attachment target portion 1203A(such as a trocar 1237) for attaching the sensor 1203 a to the endoscope802, a camera control unit (CCU) 804, a light source 805, an electricknife apparatus 806, a pneumoperitoneum apparatus 807, an ultrasonicdrive power supply 808, a VTR 809, a system controller 810, a virtualimage creating section 811, a remote controller 812A, a voice inputmicrophone 812B, a reference monitor 1213 for endoscopic live imagedisplay, a mouse 815, a keyboard 816, a virtual image display monitor1217 and an operator monitor 1232 in an operation room.

As the endoscope 802, a laparoscope is used as shown in FIGS. 85 and 86.The laparoscope has an insert portion 1237A to be inserted into anabdominal cavity of a body to be examined, a handle 1237B disposed onthe proximal end side of the insert portion 1237A, and an eye piece1237C provided at the handle 1237B. An illumination optical system andan observation optical system are provided within the insert portion1237A. The illumination optical system and the observation opticalsystem illuminate a part to be observed within an abdominal cavity of abody to be examined, and an observation image of the inside of theabdominal cavity of the body to be examined can be obtained. A lightguide connector 1202 a is provided at the handle 1237B. A connector atone end of a light guide cable having the other end connecting to thelight source apparatus is connected to the light guide connector 1202 a.Thus, illumination light from the light source apparatus 805 can beirradiated to a part to be observed through the illumination opticalsystem.

A camera head 1202A self-containing a CCD as shown in FIG. 86 isconnected to the eyepiece 1237C. A remote switch 1202B to be used forperforming an operation such as zooming in/out of an observation imageis provided in the camera head 1202A. A camera cable is extended fromthe proximal end side of the camera head 1202A. A connection connectorfor electrically connecting to the CCU 804 is provided at the other endof the camera cable.

The endoscope (laparoscope) 802 is used within the trocar 1237 (refer toFIG. 87), which is an attachment target portion for attaching the sensor1203 a, which will be described later, during surgery.

As shown in FIG. 87, the trocar 1237 has an insert portion 1237A1 to beinserted into a body cavity of a body to be examined, a body 1237B1 onthe proximal end side of the insert portion 1237A1 and an extension 1237b extending on the outer surface of the body 1237B1. The sensor 1203 ais attached onto the extension 1237 b. An air-supply connector 1207 a isprovided in the body 1237B1. A connector provided at one end of anair-supply tube having the other end connecting to the pneumoperitoneumapparatus 807 is connected to the air-supply connector 1207 a. Thus, theinside of the abdominal cavity is inflated by air supply from thepneumoperitoneum apparatus 807 so that a spatial area can be establishedfor a field of vision of and/or treatment by the endoscope 802.

The endoscope 802 is provided within the trocar 1237 having theabove-described construction and is held at the abdominal part withinthe body of a patient by the trocar 1237. By keeping this state, theinsert portion 1237A is inserted into the abdomen area. Observationimages of the inside of the abdominal cavity having been obtainedthrough the observation optical system are supplied to the CCU 804through the camera head 1202A.

As shown in FIG. 84, the CCU 804 performs signal processing on the imagepickup signals from the endoscope 802 and supplies image data (such asendoscopic live image data) based on the image pickup signals to thesystem controller 810 and the VTR 809 in an operation room. Under thecontrol of the system controller 810, image data based on a still imageor moving images of endoscopic live images is selectively output fromthe CCU 804. A detail construction of the system controller 810 will bedescribed later.

As described above, the light source apparatus 805 is a light sourceapparatus for supplying illumination light to an illumination opticalsystem provided in the endoscope 802 through a light guide within thelight guide cable.

As described above, the electric knife apparatus 806 includes a surgicaltreatment apparatus for cutting an abnormal part within the abdomen areaof a patient, for example, by using electric heat and a high-frequencyoutput apparatus for outputting high frequency current to the treatmentapparatus. The ultrasonic drive power supply 808 is a surgical treatmentapparatus for cutting or coagulating the abnormal part by using anultrasonic probe (not shown).

In addition to the above-described various kinds of equipment, thesystem controller 810 and an operator monitor 1232 are placed within anoperation room.

According to this embodiment, in order to perform treatment at aposition as shown in FIG. 83 by an operator 831 who images a body to beexamined by inserting the insert portion into the abdominal part of thepatient 830 through the trocar 1237, the operator monitor 1232 is placedat an easy-to-see position (in the direction of the field of vision)with respect to the position of the operator 831.

The operator monitor 1232 has an endoscopic image monitor 1213 a and avirtual image monitor 1217 a in parallel.

According to this embodiment, the sensor 1203 a is provided on the armof the operator 831 or the attachment target portion 1203A such as thetrocar 1237 holding the endoscope 802 therethrough in order to createand display virtual images based on a direction of field of vision ofthe endoscope 802 . The sensor 1203 a is a sensor such as a gyroscopicsensor accommodated in a unit and detects information such as an angleof insertion of the attachment target portion 1203A such as the trocar1237 into the abdomen area. The detection information of the sensor 1203a is supplied to the virtual image creating section 811, which will bedescribed later, through a connection line 1211 a. While the sensor 1203a is electrically connected to the virtual image creating section 811through the connection line 1211 a, the sensor 1203 a may be connectedto the virtual image creating section 811 in a wireless manner so as toimplement data communication. A specific construction of the attachmenttarget portion 1203A will be described later.

Though not shown, the remote controller 812A has a white balance button,a pneumoperitoneum button, a pressure button, a record button, a freezebutton, a release button, a display button, an operation button forimplementing two-dimensional display (2D display) for displaying volumerendering images, an operation button for implementing three-dimensionaldisplay (3D display) for displaying virtual images, an inserting pointbutton, a focus point button, buttons for instructing to change adisplay scale for 3D display (such as a zoom-in button and a zoom-outbutton), a display color button, a tracking button, an operation buttonfor switching and/or determining setting input information for anoperation setting mode determined by pressing one of buttons, a numerickeypad. The white balance button is used for display images displayed ona reference monitor 1213 for endoscopic live images, the virtual imagedisplay monitor 1217 or the operator monitor 1232. The pneumoperitoneumbutton is used for implementing the pneumoperitoneum apparatus 807. Thepressure button is used for increasing or decreasing the pressure forimplementing a pneumoperitoneum. The record button is used for recordingendoscopic live images in the VTR 809. The freeze button and the releasebutton are used for recording. The display button is used for displayingendoscopic live images or virtual images. The operation button for 2Ddisplay may includes an axial button, coronal button, and sagittalbutton in accordance with one of different kinds of 2D display mode. Theinserting point button is used for indicating a direction of field ofview of a virtual image displayed in a 3D display mode (and may be abutton for displaying information on insertion to the abdomen area ofthe endoscope 2 such as numerical values in X-, Y- and Z-directions ofthe abdomen area to which the endoscope 2 is inserted). The focus buttonis a button for displaying numerical values of the X-, Y- andZ-directions of a focused abdomen area. The display color button is usedfor changing a display color. The tracking button is used for tracking.The numeric keypad is used for inputting numeric values and so on.

According to this embodiment, a press-switch may be provided in a unithaving the sensor 1203 a. By pressing the switch, functions can beimplemented by manipulating buttons on the remote controller 812A.

The display I/F 821 is electrically connected to the CCU 804, the VTR809 and the reference monitor 1213. The display I/F 821 exchangesendoscopic live image data from the CCU 804 or endoscopic image datahaving been played by the VTR 809 and outputs the received endoscopiclive image data to the reference monitor 1213 and the endoscopic imagemonitor 1213 a, which will be described later, through a switchingsection 821A. Thus, the reference monitor 1213 and the endoscopic imagemonitor 1213 a display endoscopic live images based on the suppliedendoscopic live image data. In this case, the switching section 821Aswitches the output of endoscopic live image data under the switchingcontrol of the CPU 820 and outputs the endoscopic live image data to thereference monitor 1213 and/or the endoscopic image monitor 1213 a.

The reference monitor 1213 and the endoscopic image monitor 1213 a cannot only display endoscopic live images but also display settinginformation such as setting states and parameters of the apparatuses ofthe endoscope system under the display control of the CPU 820.

The CPU 820 controls different kinds of operations in the systemcontroller 810, that is, performs control over exchanges of differentkinds of signals by the communication I/F 818 and the display I/F 821,control over writing and/or reading of image data to/from the memory819, control over display by the reference monitor 13 and the endoscopicimage monitor 1213 a, and control over different kinds of operationsbased on operation signals from the remote controller 812A (or switch).

The communication I/F 826 is connected to the communication I/F 818 ofthe system controller 810 and the sensor 1203 a provided in theattachment target portion 1203A. The communication I/F 826 exchangescontrol signals required for performing different kinds of operations inconnection with the virtual image creating section 811 and the systemcontroller 810 and receives detection signals from the sensor 1203 a.The communication I/F 826 is controlled by the CPU 825, and the controlsignals are captured into the CPU 825.

The display I/F 827 outputs virtual images created under the control ofthe CPU 825 to the virtual image monitors 1217 and 1217 a through theswitching section 827A. Thus, the virtual image monitors 1217 and 1217 adisplay supplied virtual images. In this case, under the switchingcontrol of the CPU 825, the switching section 827A can switch the outputof the virtual images and output the virtual images to the selected oneof the virtual image monitors 1217 and 1217 a. When switching thedisplay of virtual images is not required, the switching section 827A isnot required. A same virtual image may be displayed on both of thevirtual image monitors 1217 and 1217 a.

The CPU 825 includes image processing means, not shown, for creating avirtual image based on a detection result from the sensor 1203 a thatthe operator 831 has by using three-dimensional image data (CT imagedata) read from the CT image DB section 823. The CPU 825 performsdisplay control for causing one of the monitors 1217 and 1217 a, whichis switched and specified by the switching section 827A, to display avirtual image created by using the image processing means in accordancewith a detection result, that is, a virtual image corresponding to anendoscopic real image.

Next, a method of attaching a sensor by using the attachment targetportion 1203A will be described with reference to FIG. 87.

According to this embodiment, as shown in FIG. 87, the sensor 1203 a isprovided at the trocar 1237 as the attachment target portion 1203A usedby the operator 831.

As described above, the trocar 1237 has the extension 1237 b extended onthe outer surface of the body 1237B1, and the sensor 1203 a is attachedonto the extension 1237 b. The sensor 1203 a may be attached on theouter surface of the body 1237B1 as indicated by the shown dotted line.Alternatively, an extension, not shown, removably fitting with the outersurface of the body 1237B1 may be provided, and the sensor 1203 a may beattached to the extension.

Therefore, by attaching the sensor 1203 a to the trocar 1237 in thisway, the direction of the insertion of the endoscope 2 within the trocar1237 substantially agrees with the direction of the insertion of thetrocar 1237. Therefore, information such as an angle of the insertion ofthe endoscope 802 can be detected by the sensor 1203 a.

According to this embodiment, the attachment target portion 1203A may bethe arm of the operator 831 as shown in FIGS. 83 and 86 instead of thetrocar 1237, and the sensor 1203 a may be attached to the arm. In thiscase, the sensor 1203 a is accommodated within a sterilized tape member1203B having a bag form and is stuck to the arm of the operator 831.Therefore, also in this case, the direction of the arm of the operator831 is similar to a scope direction (inserting direction) of theendoscope 802 within the trocar 1237 and can be matched as describedabove. Therefore, information such as an insertion angle of theendoscope 802 can be detected by the sensor 1203 a.

According to this embodiment and a first variation example in FIG. 88, aone-touch arm band 1240, which can be sterilized, may be provided as theattachment target portion 1203A, and the sensor 1203 a accommodated inthe tape member 1203B may be attached to an internal surface of the armband 1240. When the arm band 1240 itself is sterilized and has a bagform, the sensor 1203 a may be accommodated in the bag-form arm band1240 and be fixed tightly instead of accommodating in the tape member1203B.

In the first variation example, removable Velcro convex 1240 a andconcave 1240 b are provided on the both sides of the arm band 1240.Therefore, the sensor 1203 a can be attached thereto by the operator 831more easily.

According to this embodiment and a second variation example in FIG. 89,as the attachment target portion 1203A, a movable scope holder 1242 maybe used which holds the endoscope (laparoscope) 802 on the operationtable 1241.

For example, as shown in FIG. 89, the scope holder 1242 has a fixingportion 1243, a support portion 1244, a first connecting portion 1245, asecond connecting portion 1247, a slide portion 1248, a second armportion 1249 and a third connecting portion 1250. The fixing portion1243 fixes the scope holder 1242 to the operation table 1241. Thesupport portion 1244 is fixed to the fixing portion 1243. The firstconnecting portion 1245 vertically movably support a first arm portion1246 at the support portion 1244. The second connecting portion 1247 isprovided on the opposite side of the first connecting portion 1245, anda slide support portion 1247A is rotatably connected to the secondconnecting portion 1247. The slide portion 1248 can slide on the slidesupport portion 1247A. The second arm portion 1249 is strechablyprovided in the slide portion 1248. The third connecting portion 1250 isprovided on the opposite side of the slide portion 1248 and holds theendoscope 802. The third connecting portion 1250 has a holding portion1250A for holding and fixing a handle 1237B of the endoscope 802 (morespecifically, a part around the border of the insert portion 1237A andthe handle 1237B of the endoscope 802). The endoscope 802 is rotatablyheld by the holding portion 1250A.

With the scope holder 1242 according to the second variation example,the sensor 1203 a accommodated in the tape member 1203B is attachedonto, for example, the side of the third connecting portion 1250. Thus,like the configuration example of the trocar 1237 shown in FIG. 87, theinsertion direction of the endoscope 802 held by the holder 1250A cansubstantially agree with the direction of the movement of the thirdconnecting portion. Therefore, information such as an insertion angle ofthe endoscope 802 can be detected by the sensor 1203 a.

While, according to this embodiment, the sensor 1203 a is attached tothe trocar 1237, the arm of the operator 831 or the third connectingportion 1250 of the scope holder 1242, the invention is not limitedthereto. For example, the sensor 1203 a may be attached to a cap, aslipper or the like of the operator 831.

Next, an example of control over a virtual image display apparatusaccording to this embodiment will be described with reference to FIGS.90 and 91.

Here, surgery on a body to be examined within the abdomen area of apatient is performed by using an endoscope system of the virtual imagedisplay apparatus 1201 shown in FIG. 83. In this case, when theendoscope system is powered on and when the CPU 825 of the virtual imagecreating section 811 receives an instruction for virtual image displayfrom an operator through the mouse 815 or the keyboard 816, the CPU 825starts a virtual image display program recorded in a recording portion,not shown. Thus, the CPU 825 causes the monitor 1217 to display a screenrequired for displaying a virtual image.

Then, the operator inputs information on the position within the abdomenarea of the patient, for example, to which the endoscope 802 is inserted(that is, numeric value in the X, Y, and Z directions of the abdomenarea (inserting point)) by using the mouse 815 or the keyboard 816 withreference to the screen displayed on the monitor 1217. Then, similarly,the operator inputs, that is, specifies a numeric value in the axialdirection of the endoscope 802, which is being inserted into the abdomenarea (that is, a focus point).

The image processing means (not shown) creates a virtual imagecorresponding to an inserting point and a focus point of the endoscope802 based on input information. The CPU 825 displays data of the createdvirtual image on the virtual image monitor 1217 and the virtual imagemonitor 1217 a of the operator monitor 1232.

Here, endoscopic live images are displayed on the endoscopic imagemonitor 1213 a within the operator monitor 1232 for an operatorperforming surgery under the display control of the CPU 820 of thesystem controller 810. The operator 831 performs surgery with referenceto the display. In this case, the endoscope 802 is used with the sensor1203 a set in the trocar 1237 as shown in FIG. 87.

While surgery is being performed, the CPU 825 of the virtual imagecreating section 811 activates a detection program shown in FIG. 91according to this embodiment. The detection program detects an insertiondirection of the trocar 1237 or the arm of the operator 831 (ordirection of point of vision of the operator when the sensor 1203 a isattached to the cap or slipper of the operator 831) by using the sensor1203 a and displays a virtual image based on the detection result.

For example, it is assumed that, during surgery, the operator 831inclines the insert portion of the endoscope 802 with respect to theabdomen area. In this case, when endoscopic live images in accordancewith the inclination of the endoscope 802 is displayed on the referencemonitor 1213 and the endoscopic image monitor 1213 a (refer to FIG. 90),the inclination of the endoscope 802 is always detected by the sensor1203 a under the control of the CPU 825 according to this embodiment(step S121). Based on the detection result, an insertion direction ofthe trocar or the arm of the operator or a direction of point of visionof the operator is estimated, and a virtual image is created by theimage processing means within the CPU 825 (step S122). The created imageis displayed on the monitor 1217 and the virtual image monitor 1217 a(refer to FIG. 90) of the operator monitor 1232 (step S123).

Thus, since a virtual image corresponding to an endoscopic live imageupon inclination of the insert portion of the endoscope 802 can bedisplayed on the virtual image monitor 1217 a, biological imageinformation (virtual image) of a body to be examined within an observedarea of an endoscopic observation image can be obtained under endoscopicobservation.

Thus, according to this embodiment, only by attaching the sensor 1203 ato the trocar 1237 or the arm of the operator 831, a virtual imagecorresponding to an insertion angle of the endoscope 802 can bedisplayed automatically along with endoscopic live images. Therefore, anoperator can securely obtain biological image information (virtualimage) of a body to be examined within an observed area of an endoscopicobservation image while performing surgery, and the surgery can beperformed smoothly. As a result, an easy-to-use virtual displayapparatus having a simple construction can be obtained at low costs.

Since, according to this embodiment, the sensor 1203 a is provided onthe trocar 1237 holding the endoscope 802 or the arm of the operator831, the weight of the operation portion of the endoscope 802 can bereduced. Therefore, the operability of the endoscope 802 can beimproved.

Furthermore, according to this embodiment, when the sensor 1203 a isattached to a cap, a slipper or the like of the operator 831 and whenthe operator 831 moves his/her head or leg toward the direction he/sheneeds to see, the orientation (the direction of point of vision) of theoperator can be detected by the sensor 1203 a. Thus, a virtual image inaccordance with the orientation (direction of point of vision) of theoperator can be displayed under the control of the CPU 825. In otherwords, the operator 831 can display a virtual image in the directionthat the operator 831 needs to see only by moving his/her body towardthe direction that he/she needs to see. Thus, the operator 831 caneasily identify a three-dimensional, positional relationship of bloodvessels and the like behind an observed part displayed in an endoscopicimage and can securely perform a treatment.

According to this embodiment, the sensor 1203 a is provided only at thetrocar 1237 holding the endoscope 802. However, when surgery isperformed by an operator operating the endoscope 802, an operatorperforming forceps treatment by using treating devices and an assistantoperator, the sensors 1203 a may be provided at the trocars 1237 holdingthe treating devices in addition to the trocar 1237 holding theendoscope 802. Furthermore, operator monitors may be provided for theoperators, and a virtual image based on a detection result of thesensors 1203 a may be displayed.

According to this embodiment, the endoscope 802 may be an endoscopehaving a bending portion of which insert portion has the distal end thatis freely bendable. In this case, when a function for detecting abending angle of the bending portion is provided to the sensor 1203 a, avirtual image in accordance with a bending angle of the bending portioncan be displayed. When a magnetic sensor is provided in unitaccommodating the sensor 1203 a and when means for irradiating magnetismis provided to the magnetic sensor, an amount of insertion of theendoscope 802 can be detected by performing computation processing bythe CPU 825. In other words, a virtual image based on an amount ofinsertion of the endoscope 802 can be displayed. In this case, an amountof insertion of the endoscope 802 may be detected by using a rotaryencoder instead of a magnetic sensor.

With a virtual image display apparatus having a simple constructionaccording to this embodiment, biological image information of a body tobe examined within an observed area of an endoscopic observation imagecan be securely obtained at low costs under endoscopic observation,which is an advantage.

Since, with a virtual image display apparatus having a simpleconstruction according to this embodiment, biological image informationof a body to be examined within an observed area of an endoscopicobservation image can be obtained at low costs under endoscopicobservation, the virtual image display apparatus is especially effectivefor performing surgery for a case requiring further biological imageinformation of a body to be examined, which cannot be obtained fromendoscopic observation images.

Eleventh Embodiment

FIGS. 92 to 96 show a virtual image display apparatus according to aneleventh embodiment of the invention. FIG. 92 is a schematicconstruction diagram showing an entire construction of an endoscopesystem including the virtual image display apparatus. FIG. 93 is a blockdiagram showing an entire configuration of the endoscope system in FIG.92. FIG. 94 is a perspective view showing an external construction of atrocar, which is an attachment target portion to which a sensor isattached. FIGS. 95 and 96 illustrate display operations of the virtualimage display apparatus according to this embodiment. FIG. 95 is aflowchart showing main control processing by a CPU of a virtual imagecreating section. FIG. 96 is a flowchart showing voice controlprocessing by the CPU.

The same reference numerals are given to the same components as those ofthe eighth and tenth embodiment, the descriptions of which will beomitted.

As shown in FIG. 92, a virtual image display apparatus 1301 according tothis embodiment is combined with an endoscope system. More specifically,the virtual image display apparatus 1301 has an endoscope 802 asobservation means, at least two of first and second treating devices1238 and 1239 for treating a body to be examined, an attachment targetportion 1203A (such as a trocar 1237) for attaching sensors 1203 a to1203 c to the endoscope 802 and the first and second treating devices1238 and 1239, a camera control unit (CCU) 804, a light source apparatus805, an electric knife apparatus 806, a pneumoperitoneum apparatus 807,an ultrasonic drive power supply 808, a VTR 809, a system controller810, a virtual image creating section 811, a remote controller 812A, avoice input microphone 812B, a reference monitor 1213 for endoscopiclive image display, a mouse 815, a keyboard 816, a virtual image displaymonitor 1217 and three of first to third operator monitors 1232, 1234,1236 in an operation room.

In addition to these devices and apparatus, the system controller 810and the first to third operator monitors 1232, 1234 and 1236 aredisposed in an operation room.

As shown in FIG. 92, surgery under endoscopic observation may beperformed by three people including an operator operating the endoscope802, an operator performing a forceps treatment and an assistantoperator. The virtual image display apparatus 1301 according to thisembodiment is compliant with surgery by three operators.

For example, an operator performing a forceps treatment on a body to beexamined of the patient 830 by using the first treating device 1238 suchas forceps is called first operator 833. An operator operating theendoscope 802 is called second operator 831. An assistant operatorassisting the first operator by using the second treating device 1239 iscalled third operator 835. When the first to third operators 833, 831and 835 perform a treatment at a position as shown in FIG. 92, forexample, the first to third operator monitors 1232, 1234 and 1236 aredisposed at easy-to-see positions (direction of field of vision)corresponding to positions of the first to third operators 833, 831 and835.

The first operator monitor 1232 has an endoscopic image monitor 1213 aand a virtual image monitor 1217 a in parallel and is disposed at aposition, which can be seen easily by the first operator 833. The secondoperator monitor 1234 has an endoscopic image monitor 1213 b and avirtual image monitor 1217 b in parallel and is disposed at a position,which can be seen easily by the second operator 831. The third operatormonitor 1236 has an endoscopic image monitor 1213 c and a virtual imagemonitor 1217 c in parallel and is disposed at a position, which can beseen easily by the third operator 835.

According to this embodiment, the sensors 1203 a to 1203 c are attachedon the arms of the first to third operators 833, 831 and 835 or theattachment target portion 1203A such as the trocars 1237 holding theendoscope 802 and the first and second treating devices 1238 and 1239therethrough in order to create and display virtual images based on adirection of directions of insertion of the endoscope 802 and the firstand second treating devices 1238 and 1239.

The sensors 1203 a to 1203 c are sensors such as gyroscopic sensorsaccommodated in units and detect information such as an angle ofinsertion of the attachment target portion 1203A such as the trocar 1237into the abdomen area. The detection information of the sensors 1203 ato 1203 c is supplied to the virtual image creating section 811, whichwill be described later, through a connection line 1211 a. While thesensors 1203 a to 1203 c are electrically connected to the virtual imagecreating section 811 through the connection line 1211 a, the sensors1203 a to 1203 c may be connected to the virtual image creating section811 in a wireless manner so as to implement data communication.

A press-button switch 1203D to be used by an operator for, for example,implementing, changing or switching a display mode of virtual images isprovided in each of the sensors 1203 a to 1203 c (refer to FIG. 93). Aspecific construction of the attachment target portion 1203A will bedescribed later.

Though not shown, the remote controller 812A has a white balance button,a pneumoperitoneum button, a pressure button, a record button, a freezebutton, a release button, a display button, an operation button forimplementing two-dimensional display (2D display) for displaying volumerendering images, an operation button for implementing three-dimensionaldisplay (3D display) for displaying virtual images, an inserting pointbutton, a focus point button, buttons for instructing to change adisplay scale for 3D display (such as a zoom-in button and a zoom-outbutton), a display color button, a tracking button, an operation buttonfor switching and/or determining setting input information for anoperation setting mode determined by pressing one of the buttons, anumeric keypad. The white balance button is used for display imagesdisplayed on a reference monitor 1213 for endoscopic live images, thevirtual image display monitor 1217 or the operator monitors 1232, 1234and 1236. The pneumoperitoneum button is used for implementing thepneumoperitoneum apparatus 807. The pressure button is used forincreasing or decreasing the pressure for implementing apneumoperitoneum. The record button is used for recording endoscopiclive images in the VTR 809. The freeze button and the release button areused for recording. The display button is used for displaying endoscopiclive images or virtual images. The operation button for 2D display mayincludes an axial button, coronal button, and sagittal button inaccordance with one of different kinds of 2D display mode. The insertingpoint button is used for indicating a direction of field of view of avirtual image displayed in a 3D display mode (and may be a button fordisplaying information on insertion to the abdomen area of the endoscope802 such as numerical values in X-, Y- and Z-directions of the abdomenarea to which the endoscope 2 is inserted). The focus button is a buttonfor displaying numerical values of the X-, Y- and Z-directions of afocused abdomen area. The display color button is used for changing adisplay color. The tracking button is used for tracking. The numerickeypad is used for inputting numeric values and so on. By pressing theswitch 1203D provided to each of the sensors 1203 a to 1203 b (refer toFIG. 93), functions can be implemented by manipulating buttons on theremote controller 812A.

By using the remote controller 12A or the switch 1203D including thesebuttons, an operator can operate to obtain desired information promptly.

The display I/F 821 is electrically connected to the CCU 804, the VTR809 and the reference monitor 1213. The display I/F 821 exchangesendoscopic live image data from the CCU 804 or endoscopic image datahaving been played by the VTR 809 and outputs the received endoscopiclive image data to the reference monitor 1213 and the endoscopic imagemonitors 1213 a to 1213 c, which will be described later, through theswitching section 821A. Thus, the reference monitor 1213 and theendoscopic image monitors 1213 a to 1213 c display endoscopic liveimages based on the supplied endoscopic live image data. In this case,under the switching control of the CPU 820, the switching section 821Acan switch the output of the endoscopic live image data and output theendoscopic live image data to the selected one of the reference monitor1213 and endoscopic image monitors 1213 a to 1213 c.

The reference monitor 1213 and the endoscopic image monitors 1213 a to1213 c can not only display endoscopic live images but also displaysetting information such as setting states and parameters of theapparatuses of the endoscope system under the display control of the CPU820.

The CPU 820 controls different kinds of operations in the systemcontroller 810, that is, performs control over exchanges of differentkinds of signals by the communication I/F 818 and the display I/F 821,control over writing and/or reading of image data to/from the memory819, control over display by the reference monitor 1213 and theendoscopic image monitors 1213 a to 1213 c, and control over differentkinds of operations based on operation signals from the remotecontroller 812A or the switch 1203D.

The communication I/F 826 is connected to the communication I/F 818 ofthe system controller 810, the sensors 1203 a to 1203 c provided in theattachment target portions 3A for the first to third operators 833, 831and 835 and the switch 1203D. The communication I/F 826 exchangescontrol signals required for performing different kinds of operations inconnection with the virtual image creating section 811 and the systemcontroller 810, receives detection signals from the sensors 1203 a to1203 c and receives an operation signal from the switch 1203D. Thecommunication I/F 826 is controlled by the CPU 825, and the controlsignals are captured into the CPU 825.

The display I/F 827 outputs virtual images created under the control ofthe CPU 825 to the virtual image monitors 1217 and 1217 a to 1217 cthrough the switching section 827A. Thus, the virtual image monitors1217 and 1217 a to 1217 c display supplied virtual images. In this case,under the switching control of the CPU 825, the switching section 827Acan switch the output of the virtual images and output the virtualimages to the selected one of the virtual image monitors 1217 and 1217 ato 1217 c.

The CPU 825 controls different kinds of operations in the virtual imagecreating section 811, that is, performs control over exchanges ofdifferent kinds of signals by the communication I/F 826 and the displayI/F 827, control over writing and/or reading of image data to/from thememory 824, control over display by the monitors 1217 and 1217 a to 1217c, control over switching of the switching section 827A and control overdifferent kinds of operations based on operation signals from the mouse815 and/or the keyboard 816.

The CPU 825 includes image processing means, not shown, for creating avirtual image based on a detection result from the sensors 1203 a to1203 c that the first to third operators 833, 831 and 835 have by usingthree-dimensional image data (CT image data) read from the CT image DBsection 823. The CPU 825 performs display control for causing one of themonitors 1217 and 1217 a to 1217 c, which is switched and specified bythe switching section 827A, to display a virtual image created by usingthe image processing means in accordance with a detection result, thatis, a virtual image corresponding to an endoscopic real image.

Also according to this embodiment, the virtual image creating section811 may be connected to a remotely provided virtual image creatingsection, for example, through communication means so as to beconstructed as a remote surgery support system.

Next, a method of attaching a sensor by using the attachment targetportion 1203A will be described with reference to FIG. 94.

According to this embodiment, as shown in FIG. 94, the sensors 1203 a to1203 c are provided in the trocars 1237, which are the attachment targetportion 1203A used by the respective first to third operators 833, 831and 835. The trocar 1237 can be attached by holing the endoscope 2 andthe first and second treating devices 1238 and 1239 therethrough used bythe first and second operators 833 and 835.

The trocar 1237 has the extension 1237 b extended on the outer surfaceof the body 1237B1, and the sensor 1203 a (1203 b and 1203 c) having theswitch 1203D is attached onto the extension 1237 b. The sensor 1203 a(1203 b and 1203 c) may be attached on the outer surface of the body1237B1 as indicated by the shown dotted line. Alternatively, anextension, not shown, removably fitting with the outer surface of thebody 1237B1 may be provided, and the sensor 1203 a (1203 b, 1203 c) maybe attached to the extension.

Therefore, by attaching the sensor 1203 a (1203 b, 1203 c) to the trocar1237 in this way, the direction of the insertion of the endoscope 802and/or the first and second treating devices 1238 and 1239 within thetrocar 1237 substantially agrees with the direction of the insertion ofthe trocar 1237. Therefore, information such as an angle of theinsertion of the endoscope 802 and the first and second treating devices1238 and 1239 can be detected by the sensor 1203 a to 1203 c.

According to this embodiment, the attachment target portion 1203A may bethe arms of the first to third operators 833, 831 and 835 as shown inFIGS. 92 and 86 instead of the trocar 1237, and the sensors 1203 a to1203 c may be attached to the arms. In this case, each of the sensors1203 a to 1203 c is accommodated within a sterilized tape member 1203Bhaving a bag form and is stuck to the arm of each of the first to thirdoperators 833, 831 and 835. Therefore, also in this case, the directionof the arm of each of the first to third operators 833, 831 and 835 issimilar to a scope direction of the endoscope 802 within the trocar 1237or a direction of insertion of one of the first and second treatingdevices 1238 and 1239 and can be matched as described above. Therefore,information such as an insertion angle of the endoscope 802 and thefirst and second treating devices 1238 and 1239 can be detected by thesensors 1203 a to 1203 c.

Also in this embodiment, like the tenth embodiment, the attachmenttarget portion 1203A may be the place in the variation example in FIG.88 or the variation example in FIG. 89.

By the way, by manipulating the switch 1203D (or the remote controller812A) provided in the sensors 1203 a (1203 b, 1203 c) in the virtualimage display apparatus according to this embodiment, a display mode forvirtual images can be selected, implemented or switched.

For example, one of the first to third operators 833, 831 and 835 canselect and implement a display mode for virtual display image byproperly pressing the switch 1203D (refer to FIG. 93).

Next, a control example of the virtual image display apparatus accordingto this embodiment to be implemented by a switching operation will bedescribed with reference to FIG. 95.

First of all, a basic operation of the virtual image display apparatusaccording to this embodiment will be described.

Here, an operator on a body to be examined within the abdomen area of apatient is performed by using an endoscope system of the virtual imagedisplay apparatus 1301 shown in FIG. 92. In this case, when theendoscope system is powered on and when the CPU 825 of the virtual imagecreating section 811 receives an instruction for virtual image displayfrom an operator through the mouse 815 or the keyboard 816, the CPU 825starts a virtual image display program recorded in a recording portion,not shown. Thus, the CPU 825 causes the monitor 1217 to display a screenrequired for displaying a virtual image.

Then, the operator inputs information on the position within the abdomenarea of the patient, for example, to which the endoscope 802 is inserted(that is, numeric value in the X, Y, and Z directions of the abdomenarea (inserting point)) by using the mouse 815 or the keyboard 816 withreference to the screen displayed on the monitor 1217. Then, similarly,the operator inputs a numeric value in the axial direction of theendoscope 802, which is being inserted into the abdomen area (that is,focus point). Also for the first and second treating devices 1238 and1239, respective required information are input with reference toscreens, not shown.

The image processing means (not shown) creates virtual imagescorresponding to an inserting point and focus point of the endoscope 802and inserting points and focus points of the first and second treatingdevices 1238 and 1239 based on input information. The CPU 825 displaysdata of the created virtual images on the virtual image monitor 1217 andthe first to third operator monitors 1232, 1234 and 1236. In this case,virtual images corresponding to the endoscope 802 are mainly displayedon the virtual image monitor 1217. In addition, virtual imagescorresponding to the first and second treating devices 1238 and 1239 maybe selected and be displayed.

Here, endoscopic live images are displayed on the endoscopic imagemonitors 1213 a to 1213 c within the first to third operator monitors1232, 1234 and 1236 for the first to third operators performing surgeryunder the display control of the CPU 820 of the system controller 810.The first to third operators 833, 831 and 835 perform surgery withreference to the display. In this case, the endoscope 802 and the firstand second treating devices 1238 and 1239 are used with the sensors 1203a to 1203 c set in the trocars 1237 as shown in FIG. 94. The sensors1203 a to 1203 c may be attached onto the arms of the respectiveoperators through the tape portions 1203B instead of the trocars 1237.

While surgery is being performed, the CPU 825 of the virtual imagecreating section 811 according to this embodiment creates a virtualimage in accordance with endoscopic live images and based on a detectionresult from the sensor 1203 a of the endoscope 802 by means of the imageprocessing means within the CPU 825. Then, the CPU 825 causes thevirtual image monitors 1217 b of the monitor 1217 and the secondoperator monitor 1234 to display the created image. At the same time,the CPU 825 creates virtual images by means of the image processingmeans within the CPU 825 based on detection result from the sensors 1203b and 1203 c of the first and second treating devices 1238 and 1239 andcauses the virtual image monitors 1217 a and 1217 c of the first andthird operator monitors 1232 and 1236 to display the created images.

For example, it is assumed that, during surgery, the second operator 831inclines the insert portion of the endoscope 802 with respect to theabdomen area. In this case, when endoscopic live images in accordancewith the inclination of the endoscope 802 is displayed on the referencemonitor 1213 and the endoscopic image monitors 1213 a to 1213 c, theinclination of the endoscope 802 is always detected by the sensor 1203 aaccording to this embodiment. The CPU 825 creates a virtual image basedon the detection result by means of the image processing means withinthe CPU 825. The CPU 825 causes the monitor 1217 and the virtual imagemonitor 1217 b of the second operator monitor 1234 to display thecreated image. Similarly, for the first and second treating devices 1238and 1239, the CPU 825 creates virtual images based on detection resultsfrom the sensors 1203 b and 1203 c by means of the image processingmeans within the CPU 825. The CPU 825 causes the virtual image monitors1217 a and 1217 c of the first and third operator monitors 1232 and 1236to display the created images.

Thus, since virtual images corresponding to endoscopic live images uponinclination of the insert portion of the endoscope 802 and/or the firstand second treating devices 1238 and 1239 can be displayed on thevirtual image monitors 1217 a to 1217 b, the first to third operators833, 831 and 835 can obtain biological image information of a body to beexamined within an observed area of an endoscopic observation imageunder endoscopic observation.

In the control example according to this embodiment, the CPU 825activates a detection program shown in FIG. 95 during display of virtualimages. In response to a change instruction request of a display modefrom an operator through the switch 1203D during surgery, the detectionprogram detects the operation signal and displays a virtual image basedon the detection result.

The CPU 825 always detects the presence of a switch manipulation on theswitch 1203D in judgment processing at a step S131. In this case, if itis judged that a switch manipulation has been performed on the switch1203D, the CPU 825 identifies the switch 1203D pressed in the processingat a subsequent step S132 (that is, the switch 1203D pressed by one ofthe first to third operators 833, 831 and 835) and the type of theoperation instruction (command), and the processing goes to judgmentprocessing at a step S133. On the other hand, if it is judged that noswitch operation has been performed, the CPU 825 continuously performsjudgment processing until a switch manipulation is performed on theswitch 1203D.

Then, the CPU 825 judges whether or not the type of the operationinstruction (command) by the switch 1203D, which is recognized in thejudgment processing at the step S133, is for the simultaneous displaymode. If not, the processing returns to the step S131. If for thesimultaneous display mode, the processing goes to a step S134.

Then, in the processing at the step S134, the CPU 825 performs displayswitching processing based on the type of the operation instruction(command) at the step S132. In other words, since the type of theoperation instruction (command) by the switch 1203D is a command for thesimultaneous display mode, the CPU 825 controls, in the processing atthe step S134, the switching section 827A shown in FIG. 93 such that avirtual image corresponding to the treating device of one of the firstto third operators 833, 831 and 835 having pressed the switch 1203D canbe output and displayed on the virtual image monitors 1217 a to 1217 cof the first to third operator monitors 1232, 1234 and 1236. Thus, thevirtual image corresponding to the treating device (such as one treatingdevice of the endoscope 802 and the first and second treating devices1238 and 1239) of the operator having pressed the switch 1203D can besimultaneously displayed on the virtual image monitors 1217 a to 1217 cdisposed in the directions of the fields of vision of the operators.

The virtual image display apparatus 1301 according to this embodimentcan select and execute a virtual display mode not only through theswitch 1203D but also by voice of an operator. The control example byvoice input will be described with reference to FIG. 96. The voice inputmicrophone 812B shown in FIGS. 92 and 93 are used by all of the first tothird operators 833, 831 and 835.

The CPU 825 activates the detection program shown in FIG. 96 duringdisplay of virtual images based on detection results from the sensors1203 a to 1203 c of the endoscope 802 and the first and second treatingdevices 1238 and 1239. In response to a change instruction request of adisplay mode from an operator through the voice input microphone 812Bduring surgery, the detection program detects the type of the voiceinstruction and displays a virtual image based on the detection result.

The CPU 825 exchanges signals with the communication I/F 818 of thesystem controller 810 and always detects the presence of a voice inputthrough the voice input microphone 812B in judgment processing at a stepS141. In this case, if it is judged that a voice input instructionthrough the voice input microphone 812B has been performed, the CPU 825identifies the voice input microphone 812B input in the processing at asubsequent step S142 (that is, the voice input microphone 812B of one ofthe first to third operators 833, 831 and 835) and the type of the voiceinstruction (command), and the processing goes to judgment processing ata step S143. On the other hand, if it is judged that no voice inputinstruction has been performed, the CPU 825 continuously performsjudgment processing until a voice input instruction is performed throughthe voice input microphone 812B.

Then, the CPU 825 judges whether or not the type of the voiceinstruction (command) through the voice input microphone 812B, which isrecognized in the judgment processing at the step S143, is a switchingoperation command. If not, the processing returns to the step S141. Ifthe command is for a switching operation, the processing goes to a stepS144.

Then, since the type of voice instruction (command) through the voiceinput microphone 812B is a command for a switching operation, the CPU825 controls to perform virtual image display based on the type of thevoice instruction. For example, the CPU 825 controls the switchingsection 827A shown in FIG. 93 such that a virtual image corresponding tothe treating device of one of the first to third operators 833, 831 and835 having given the voice instruction through the voice inputmicrophone 812B can be output and displayed on the virtual imagemonitors 1217 a to 1217 c of the first to third operator monitors 1232,1234 ad 1236. Thus, the virtual image corresponding to the treatingdevice (such as one treating device of the endoscope 802 and the firstand second treating devices 1238 and 1239) of the operator who inputsthe voice instruction by using the voice input microphone 812B can besimultaneously displayed on the virtual image monitors 1217 a to 1217 cdisposed in the directions of the fields of vision of the operators.

Therefore, according to this embodiment, only by performing a switchingoperation through the switch 1203D or inputting a voice instructionthrough the voice input microphone 812B, a virtual image correspondingto the treating device (such as one treating device of the endoscope 802and the first and second treating devices 1238 and 1239) of the operatorhaving pressed the switch 1203D or input the voice instruction can besimultaneously displayed on the virtual image monitors 1217 a to 1217 cdisposed in the directions of the fields of vision of the operators.Therefore, an operator can securely obtain biological image information(virtual image) of a body to be examined within an observed area of anendoscopic observation image while performing surgery, and the surgerycan be performed smoothly. As a result, an easy-to-use virtual displayapparatus having a simple construction can be obtained at low costs.

Since, according to this embodiment, the sensors 1203 a are provided onthe trocars 1237 holding the endoscope 802 and the first and secondtreating devices 1238 and 1239 or the arms of the first to thirdoperators, the weight of the endoscope 802 and first and second treatingdevices 1238 and 1239 can be reduced. Therefore, the operability ofthese apparatuses can be improved.

Twelfth Embodiment

FIG. 97 relates to a virtual image display apparatus of a twelfthembodiment and is a flowchart showing control processing by a CPU of avirtual image creating section. In FIG. 97, the same reference numeralsare given to the same steps S as those of the processing shown in FIG.95 of the eleventh embodiment.

The virtual image display apparatus according to this embodiment issubstantially the same as the virtual image display apparatus accordingto the eleventh embodiment, but virtual display control processing bythe CPU 825 is different.

A control example by the virtual image display apparatus according tothis embodiment will be described with reference to FIG. 97.

Like the eleventh embodiment, the CPU 825 activates a processing routineshown in FIG. 97 during display of virtual images based on detectionresults from the sensors 1203 a to 1203 c of the endoscope 802 and thefirst and second treating devices 1238 and 1239 and always detects thepresence of a switch manipulation on the switch 1203D in judgmentprocessing at a step S131. In this case, if it is judged that a switchmanipulation has been performed on the switch 1203D, the CPU 825identifies the switch 1203 pressed in the processing at a subsequentstep S132 (that is, the switch 1203D pressed by one of the first tothird operators 833, 831 and 835) and the type of the operationinstruction (command), and the processing goes to judgment processing ata step S145. On the other hand, if it is judged that no switch operationhas been performed, the CPU 825 continuously performs judgmentprocessing until a switch manipulation is performed on the switch 1203D.

Then, the CPU 825 judges whether or not the type of the operationinstruction (command) by the switch 1203D, which is recognized in thejudgment processing at the step S145, is for a different display mode.If not, the processing returns to the step S131. If for a differentdisplay mode, the processing goes to a step S134.

Then, in the processing at the step S134, the CPU 825 performs displayswitching processing based on the type of the operation instruction(command) at the step S132. In other words, since the type of theoperation instruction (command) by the switch 1203D is a command for adifferent display mode, the CPU 825 controls, in the processing at thestep S134, the switching section 827A shown in FIG. 93 such that virtualimages corresponding to the treating devices used by the first to thirdoperators 833, 831 and 835 can be output and displayed on the virtualimage monitors 1217 a to 1217 c of the first to third operator monitors1232, 1234 and 1236 irrespective of one of the first to third operators833, 831 and 835 having pressed the switch 1203D. Thus, the virtualimages corresponding to the operators' treating devices (such as theendoscope 802 and the first and second treating devices 1238 and 1239)in accordance with the directions of the fields of vision of theoperators can be displayed on the virtual image monitors 1217 a to 1217c disposed in the directions of the fields of vision of the operators.

Also according to this embodiment, virtual display control may beperformed by using the voice input microphone 812B like the eleventhembodiment.

Therefore, according to this embodiment, only by performing a switchingoperation through the switch 1203D or inputting a voice instructionthrough the voice input microphone 812B, virtual images corresponding totreating devices (such as the endoscope 802 and the first and secondtreating devices 1238 and 1239) of the operator can be separatelydisplayed on the virtual image monitors 1217 a to 1217 c disposed in thedirections of the fields of vision of the operators. Therefore, surgerycan be performed smoothly. The other advantages are the same as those ofthe eleventh embodiment.

In the eleventh and twelfth embodiments according to the invention, theendoscope 802 may be an endoscope having a bending portion, the distalend of the insert portion of which is freely bendable. In this case,when a function for detecting a bending angle of the bending portion isprovided to the sensor 1203 a, a virtual image in accordance with abending angle of the bending portion can be displayed. When a magneticsensor is provided in a unit accommodating the sensor 1203 a and whenmeans for irradiating magnetism is provided to the magnetic sensor, anamount of insertion of the endoscope 802 can be detected by performingcomputation processing by the CPU 825. In other words, virtual imagesbased on amounts of insertion of the endoscope 802 and first and secondtreating devices 1238 and 1239 can be displayed. In this case, amountsof insertion of the endoscope 802 and first and second treating devices1238 and 1239 may be detected by using a rotary encoder instead of amagnetic sensor.

With a virtual image display apparatus having a simple constructionaccording to this embodiment, biological image information of a body tobe examined within an observed area of an endoscopic observation imagecan be obtained at low costs under endoscopic observation and besecurely provided to multiple operators during surgery as required,which is an advantage.

Since, with a virtual image display apparatus having a simpleconstruction according to this embodiment, biological image informationof a body to be examined within an observed area of an endoscopicobservation image can be obtained at low costs under endoscopicobservation and be securely provided to multiple operators duringsurgery as required, the virtual image display apparatus is especiallyeffective for performing surgery by an operator operating an endoscopeand an operator and assistant performing a forceps treatment by usingthe first and second treating devices.

Thirteenth Embodiment

FIGS. 98 to 111 relate to a thirteenth embodiment of the invention. FIG.98 is an entire configuration diagram showing an object observationsystem according to a thirteenth embodiment. FIG. 99 is a constructiondiagram showing a construction of a remote controller for an operator inFIG. 98. FIG. 100 is a screen display example of a virtual image displayscreen displayed on a VR monitor in FIG. 98. FIG. 101 is a screendisplay example on which a virtual image is displayed in a virtual imagedisplay area in FIG. 100. FIG. 102 is an example of an endoscopic liveimage displayed on an endoscope monitor in FIG. 98. FIG. 103 is anexample of an endoscopic live image displayed on the endoscope monitorwhen the endoscope is moved. FIG. 104 is a screen display example inwhich a virtual image agreeing with the endoscopic live image in FIG.103 is displayed on the virtual image display area. FIG. 105 is aflowchart showing a processing operation, which is a feature of thethirteenth embodiment. FIG. 106 is an example of an endoscopic liveimage for illustrating an operation of this embodiment. FIG. 107 is afirst screen display example of a virtual image display screen forillustrating the operation of this embodiment. FIG. 108 is a screendisplay example of a virtual image display screen on which a virtualimage in FIG. 107 is enlarged. FIG. 109 is a second screen displayexample of a virtual image display screen for illustrating an operationof this embodiment. FIG. 110 is a screen display example of a virtualimage display screen when organ removal processing is performed on thevirtual image in FIG. 108. FIG. 111 is an entire configuration diagramof an object observation system showing a variation example of thethirteenth embodiment.

According to this embodiment, the invention is applied to a surgicalsystem for an endoscopic surgery.

As shown in FIG. 98, an object observation system 1401 according to thethirteenth embodiment includes an endoscope 1402, a light sourceapparatus 1403, a camera head 1404, a camera control unit (CCU) 1405, anendoscope monitor 1406, a virtual image creating section 1407, a volumerendering (VR, hereinafter) monitor 1408, multiple medical equipment1409 and a system controller 1410. The endoscope 1402 is observationmeans which can observe a body to be examined. The light sourceapparatus 1403 supplies illumination light to the endoscope 1402. Thecamera head 1404 is removably attached to the endoscope 1402 andself-contains an image pickup apparatus for picking up a body to beexamined image obtained by the endoscope 1402. The CCU 1405 performssignal processing on the image pickup apparatus of the camera head 1404.The endoscope monitor 1406 displays endoscopic optical image resultingfrom signal processing by the CCU 1405 as endoscopic live images. Thevirtual image creating section 1407 performs image processing onpre-stored virtual image data and creates a volume rendering image(simply called virtual image, hereinafter). The VR monitor 1408 displaysa virtual image resulting from image processing by the virtual imagecreating section 1407 as a reference image. The multiple medicalequipment 1409 performs a treatment on an affected part of a patient,which is a body to be examined. The system controller 1410 centrallycontrols the light source apparatus 1403, the CCU 1405, the virtualimage creating section 1407 and the medical equipment 1409.

Though not shown, the endoscope 1402 has a long and narrow insertportion and an eyepiece connected to the proximal end of the insertportion. The endoscope 1402 holds a light guide (not shown) therethroughfor transmitting illumination light. The light guide transmitsillumination light from the light source apparatus 1403. Theillumination light having been transmitted from the light guideilluminates a body to be examined such as an affected part from anillumination optical system (not shown) disposed at the distal end ofthe insert portion.

The endoscope 1402 captures a body to be examined image from anobjective optical system (not shown) adjacent to the illuminationoptical system. The captured subject image is transmitted to theeyepiece by an image transmitting optical system (not shown) such as arelay lens and an image guide and is enlarged from an eyepiece opticalsystem (not shown) in the eyepiece so that the body to be examined imagecan be observed as an endoscopic optical image.

According to this embodiment, the endoscope 1402 includes an inclinationangle sensor 1411 for detecting an inclination angle of the insertportion. Inclination angle data detected by the inclination angle sensor1411 is supplied to the virtual image creating section 1407. By startingtracking, which will be described later, the virtual image creatingsection 1407 performs image processing on virtual image data based oninclination angle data detected by the inclination angle sensor 1411such that the result can agree with endoscopic live images.

The camera head 1404 removably attached to the endoscope eyepiece cancapture an endoscopic optical image transmitted from the eyepieceoptical system of the endoscope eyepiece. The camera head 1404optoelectronically converts the endoscopic optical image captured fromthe endoscope 1402 to image pickup signals by means of an image pickupapparatus (not shown) such as a CCD and outputs the image pickup signalsto the CCU 1405.

The CCU 1405 performs signal processing on image pickup signals from thecamera head 1404 and generates standard video signals thereby. Then, theCCU 1405 outputs the standard video signals to the endoscope monitor1406 through the system controller 1410. The endoscope monitor 1406displays an endoscopic optical image on the display screen as anendoscopic live image.

While the object observation system 1401 according to this embodimenthas an optical endoscope which can observe, through the eyepiece, a bodyto be examined image captured from the distal end of the insert portionand transmitted by image transmitting means to the eyepiece and a camerahead, which is mounted at the eyepiece of the optical endoscope, forpicking up an endoscopic optical image from the eyepiece, the inventionis not limited thereto. The object observation system 1401 may includean electronic endoscope self-containing, at the distal end of the insertportion, an image pickup apparatus for picking up a body to be examinedimage. In this case, the electronic endoscope may have a scalingfunction by which an objective optical system can be moved in theoptical axis direction.

The CCU 1405 supplies generated video signals to the VTR 1412. The VTR1412 is connected to the system controller 1410 and records and stores adesired endoscopic optical image in response to an operation instructionfrom an operator.

The medical equipment 1409 includes a pneumoperitoneum apparatus 1409 a,an electric knife apparatus 1409 b, and an ultrasonic surgical apparatus1409 c. The pneumoperitoneum apparatus 1409 a supplies gas such ascarbon dioxide into the abdomen area of a patient through apneumoperitoneum tube (not shown) in order to establish a field ofvision within the abdomen area. The electric knife apparatus 1409 bperforms coagulation/resection treatments on an affected part bysupplying high frequency power to an electric knife (not shown). Theultrasonic surgical apparatus 1409 c performs coagulation/resectiontreatments on an affected part by supplying electric energy to anultrasonic treating device (not shown) and using ultrasonic vibrationgenerated by the ultrasonic treating device.

These medical equipment 1409 are connected to the system controller1410.

The system controller 1410 centrally controls different kinds ofoperations of the entire system. The system controller 1410 has acommunication interface (called communication I/F, hereinafter) 1413, amemory 1414, a CPU (central processing unit) 1415 as a control portionand a display interface (called display I/F, hereinafter) 1416.

The communication I/F 1413 communicates with the light source apparatus1403, the CCU 1405, the virtual image creating section 1407 and themedical equipment 1409. The exchange of control signals and the exchangeof image data are controlled by the CPU 1415. A remote controller 1417as virtual image change instruction means is connected to thecommunication I/F 1413. The remote controller 1417 is used by anoperator to instruct to perform image processing on a virtual imagedisplayed on the VR monitor 1408 as described later. A detailconstruction of the remote controller 1417 will be described later.

The memory 1414 stores image data of endoscopic still images and datasuch as equipment setting information, for example. The data storing andreading are controlled by the CPU 1415.

The display I/F 1416 outputs video signals from the CCU 1405 or the VTR1412 to the endoscope monitor 1406. Thus, an endoscopic live image canbe displayed on a display screen of the endoscope monitor 1406.

The CPU 1415 controls different kinds of operations in the systemcontroller 1410, that is, performs control over exchanges of differentkinds of signals by the communication I/F 1413 and the display I/F 1416,control over writing and/or reading of image data to/from the memory1414, control over display by the endoscope monitor 1406, and controlover different kinds of operations based on operation instructionsignals from the remote controller 1417.

The system controller 1410 controls the medical equipment 1409 under thecontrol of the CPU 1415. The system controller 1410 outputs videosignals from the CCU 1405 to the endoscope monitor 1406. Thus,endoscopic live images can be displayed on a display screen of theendoscope monitor 1406.

In the system controller 1410, the CPU 1415 controls the virtual imagecreating section 1407 based on an operation instruction signal from theremote controller 1417.

The virtual image creating section 1407 has a CT image DB section 1418,a memory 1419, a CPU 1420, a communication I/F 1421 and a display I/F1422.

The CT image DB section 1418 includes a CT image data capturing portion(not shown) for capturing virtual image data created by a publicly knownCT apparatus, not shown, for imaging an X-ray tomographic image of abody to be examined through a portable memory medium such as amagneto-optical (MO) disk and a digital versatile disk (DVD). Thus, theCT image DB section 1418 can store the captured virtual image data. Thatis, the CT image DB section 1418 includes virtual image data storingmeans. The reading and writing of the virtual image data from/to the CTimage DB section 1418 are controlled by the CPU 1420.

The memory 1419 stores the virtual image data from a portable recordingmedium and data such as virtual image data image-processed by the CPU1420. Thus, the storing and reading data are controlled by the CPU 1420.

The communication I/F 1421 is connected to the communication I/F 1413 ofthe system controller 1410 and the inclination angle sensor 1411. Thecommunication I/F 1421 exchanges control signals required for performingdifferent kinds of operations in connection with the virtual imagecreating section 1407 and the system controller 1410. The communicationI/F 1421 is controlled by the CPU 1420, and the received signals arecaptured into the CPU 1420.

The display I/F 1422 sends virtual image data created under the controlof the CPU 1420 to the VR monitor 1408. Thus, a virtual image isdisplayed on the VR monitor 1408 connecting to the display I/F 1422.

The mouse 1423 and the keyboard 1424 are connected to the CPU 1420. Themouse 1423 and the keyboard 1424 are operation means to be used forinputting and/or setting different kinds of setting information. Asdescribed later, the mouse 1423 and the keyboard 1424 may be used asobservation information input means to input inserting point informationand focus point information of the endoscope 1402 with respect to a bodyto be examined.

The CPU 1420 performs different kinds of operations in the virtual imagecreating section 1407, that is, performs control over exchangesdifferent kinds of signals by the communication I/F 1421 and the displayI/F 1422, control over writing and/or reading of image data to/from thememory 1419, control over display by the VR monitor 1408, and controlover different kinds of operations based on operation signals from themouse 1423 and/or the keyboard 1424.

The CPU 1420 performs display control such that image processing can beperformed on virtual image data read from the CT image DB section 1418based on inclination angle data from the inclination angle sensor 1411and the virtual image can be displayed on the VR monitor 1408.

The CPU 1420 further performs virtual image change processing forchanging a virtual image based on an operation instruction from theremote controller 1417 for a virtual image displayed on the VR monitor1408 under the control of the CPU 1415 of the system controller 1410. Inother words, the CPU 1415 of the system controller 1410 and the CPU 1420of the virtual image creating section 1407 are included in virtual imageprocessing means.

The remote controller 1417 includes, as shown in FIG. 99, an imagechange operation portion 1431 for performing different kinds of imagechange processing and a tracking button 1432 for implementing tracking,for example.

The image change operation portion 1431 includes, as image changecommands, a zoom-out button 1431 a, a zoom-in button 1431 b, a displaycolor button 1431 c, a highlight button 1431 d and a remove organ button1431 e. The zoom-out button 1431 a is used for decreasing a displayscale. The zoom-in button 1431 b is used for increasing a display scale.The display color button 1431 c is used for changing a display color ofa predetermined area. The highlight button 1431 d is used forhighlighting a predetermined area by increasing or decreasing theintensity. The remove organ button 1431 e is used for removing an organso as to view a predetermined area easily.

By using the remote controller 1417 having these image change commands(buttons 1431 a to 1431 e), an operator can perform operations forobtaining a desired virtual image.

Next, a display example, which is a feature of the object observationsystem 1401, will be described with reference to FIGS. 100 to 104.

In response to an operation instruction through a manipulation on theremote controller 1417 by an operator, the CPU 1415 of the systemcontroller 1410 controls the CPU 1420 of the virtual image creatingportion 1407 to display a virtual image display screen 1440 shown inFIG. 100, for example, on the display screen of the VR monitor 1408.

The virtual image display screen 1440 includes a virtual image displayarea 1441, a 2D image display area 1442, an operation setting area 1443and a selected display area 1444. The virtual image display area 1441 isthe center of the screen and displays a virtual image. The 2D imagedisplay area 1442 is a part close to the left end of the screen anddisplays multiple 2D images. The operation setting area 1443 is a partclose to the right end of the screen and is used for manipulating and/orsetting the virtual image display area 1441. The selected display area1444 is disposed in a part close to the lowest end of the screen and isused for implementing 3D display of one of the other multiple referenceimages (thumbnail images).

The operation setting area 1443 includes an inserting point input area1445, and a focus-point input area 1446. The inserting point input area1445 is used for inputting values (called inserting point) in the X, Yand Z directions of the abdomen area into which the endoscope 1402 isinserted. The focus-point input area 1446 is used for inputting values(in angle, called focus point) in the X, Y and Z directions of the axialdirection of the endoscope 1402 where the endoscope 1402 is insertedinto the abdomen area.

In accordance with inputs to these inserting point input area 1445 andfocus point input area 1446, the CPU 1420 of the virtual image creatingsection 1407 determines a direction of line of vision of a virtual imagein order to implement virtual image display.

The operation setting area 1443 includes a zoom-in/zoom out operationarea 1447 and a tracking start/stop button 1448. The zoom-in/zoom outarea 1447 includes a zoom-in switch 1447 a and zoom-out switch 1447 bfor increasing and decreasing a display scale. The tracking start/stopbutton 1448 is used for starting/stopping tracking.

In order to activate the object observation system 1401, the virtualimage display screen 1440 shown in FIG. 100 is displayed on the VRmonitor 1408 first of all. Then, information (inserting point)indicating into which point of the abdomen area of a patient theendoscope 1402 is to be inserted is input to the inserting point inputarea 1445 by using the mouse 1423 or the keyboard 1424. After that, thefocus-point input area 1446 is selected, and a value (focus point) inthe axial direction of the endoscope 1402 is required to input in asimilar way the focus point input area 1446 where the endoscope 1402 isinserted into the abdomen area.

In other words, the CPU 1420 of the virtual image creating section 1407determines a direction of line of vision based on positional information(inserting point and focus point) of the endoscope 1402, performs imageprocessing on virtual image data and displays the virtual image on thevirtual display area 1441.

Thus, as shown in FIG. 101, a display screen displaying a virtual imagein response to an input of positional information (inserting point andfocus point) of a predetermined endoscope is obtained on the virtualimage display area 1441. Here, an endoscopic live image is displayed onthe endoscope monitor 1406 as shown in FIG. 102.

Upon starting tracking, endoscopic live images are displayed on theendoscope monitor 1406 in response to movement of the endoscope as shownin FIG. 103, for example. Thus, based on inclination angle data detectedby the inclination angle sensor 1411, the CPU 1420 of the virtual imagecreating section performs image processing on virtual image data inaccordance with the endoscopic live images and display the virtual imageon the virtual image display area 1441 as shown in FIG. 104.

According to this embodiment, based on an operation instruction by anoperator during surgery through the remote controller 1417, image changeprocessing can be implemented such as zooming-in, zooming-out and organremoval.

A processing operation, which is a feature of this embodiment, will bedescribed in detail with reference to FIGS. 106 to 109 based on aflowchart shown in FIG. 105.

Here, surgery is performed on a body to be examined within the abdomenarea of a patient by using the object observation system 1401 shown inFIG. 98. In this case, when the object observation system 1401 has powerapplied thereto, a program based on a control method for the objectobservation system of the invention, which is stored in the CPU 1415 ofthe system controller 1410, is started first of all. Thus, the CPU 1415of the system controller 1410 controls the CPU 1420 of the virtual imagecreating section 1407. As described above, the virtual image displayscreen 1440 shown in FIG. 100 is displayed on the VR monitor 1408.

Then, by using the mouse 1423 or the keyboard 1424 and with reference toa virtual image displayed on the virtual image display area 1441 of theVR monitor 1408, a nurse or an operator inputs, in the inserting pointinput area 1445, information (inserting point) regarding which positionin the abdomen area of a patient the endoscope 1402 is inserted into(step S151). Then, the nurse or operator selects the focus-point inputarea 1446 and inputs an axial value (focus point) of the endoscope 1402thereto where the endoscope 1402 is inserted to the abdomen areasimilarly in the focus-point input area 1446 (step S152). Thus, thedirection of the line of vision is determined (step S153). The stepsS151 and S152 are included in an observation information input process.

Hence, the virtual image data in accordance with the inserting point andfocus point of the endoscope 1402 undergoes image processing by the CPU1420 of the virtual image creating section 1407. Then, the result isdisplayed in the virtual image display area 1441 of the virtual imagedisplay screen 1440 as shown in FIG. 101, for example.

Then, the operator inserts the endoscope 1402 into the abdomen area ofthe patient. In a body to be examined image obtaining process, theobject observation system 1401 causes the endoscopic live imagesobtained by the endoscope 1402 to be displayed on the display screen ofthe endoscope monitor 1406 under the display control of the CPU 1415 ofthe system controller 1410 as shown in FIG. 102, for example.

The operator performs surgery with reference to the endoscopic liveimages and sometimes with reference to the virtual image display screen1440.

Then, the operator starts tracking by pressing the tracking button 1432of the remote controller 1417 (step S155).

Thus, the CPU 1420 of the virtual image creating section 1407 measuresan attitude angle (step S156) by always detecting an inclination of theendoscope 1402 by using the inclination angle sensor 1411 and determineswhether the attitude angle is changed or not (step S157).

Here, during surgery, an operator moves the endoscope 1402. Then,endoscopic live images in accordance with the inclinations of theendoscope 1402 are displayed as shown in FIG. 102, for example, on theendoscope monitor 1406.

On the other hand, when the CPU 1420 of the virtual image creatingsection 1407 determines the attitude angle has been changed here, theCPU 1420 determines a direction of a line of vision (focus point) of theendoscope 1402 based on the detected inclination angle data (step S157).Then, the CPU 1420 of the virtual image creating section 1407 performsimage processing on the virtual image data such that the virtual imagescan agree with the endoscopic live images, creates virtual images (stepS159) and causes the VR monitor 1408 (in the virtual image display area1441 of the virtual display screen 1440) to display the virtual images.In other words, the step S159 is a virtual image processing process.

Here, when the endoscope is an electronic endoscope having a scalingfunction, the display scale of virtual images may be changed such thatthe virtual images can agree with the endoscopic live images, which arescaled in accordance with a scaling operation of the electronicendoscope, in the virtual image processing process.

Thus, the virtual images, as shown in FIG. 103, corresponding to theendoscopic live images in accordance with different states of theposition, direction, display scale and so on of the endoscope can bedisplayed on the virtual display screen 1440 of the VR monitor 1408. Theoperator can obtain more detail image information fast and securelythereby.

Based on an operation instruction signal by the operator through theremote controller 1417, the CPU 1415 of the system controller 1410detects whether an image change command is input or not (step S160). Ifso, the CPU 1415 controls the CPU 1420 of the virtual image creatingsection 1407 to perform image change processing in accordance with thecommand (step S161). In other words, the step S161 is included in avirtual image change process.

Here, for example, as shown in FIG. 106, an endoscopic live image isdisplayed on the display screen of the endoscope monitor 1406, and thevirtual display screen 1440 is displayed on the display screen of the VRmonitor 1408 as shown in FIG. 107.

In this case, the operator manipulates the zoom-in button 1431 b of theremote controller 1417 in order to increase the display scale of thevirtual image displayed on the virtual image display area 1441. Thus,the CPU 1420 of the virtual image creating section 1407 performs zoom-inprocessing on the virtual image currently displayed on the virtual imagedisplay area 1441 in accordance with the manipulation on the zoom-inbutton 1431 b of the remote controller 1417 and causes the virtual imageto be displayed on the virtual image display area 1441 as shown in FIG.108.

When the virtual display screen 1440 is displayed as shown in FIG. 109,the operator may manipulate the remove organ button 1431 e of the remotecontroller 1417 in order to check how the blood vessels lie by gettingthe organ out of the virtual image.

Thus, the CPU 1420 of the virtual image creating section 1407 performsorgan removal processing on the virtual image currently displayed on thevirtual image display area 1441 in accordance with the manipulation onthe organ remove button 1431 e of the remote controller 1417 and causesthe virtual image to be displayed on the virtual image display area 1441as shown in FIG. 110.

The virtual display screen 1440 shown in FIGS. 109 and 110 has thevirtual image display area 1441 extended to the right end without theoperation setting area 1443.

According to this embodiment, a virtual image can be changed byzooming-in, zooming-out, removing organs and/or the like based on anoperation instruction through the remote controller 1417 by an operatorduring surgery.

Subsequently, the processing from the step S156 is repeated until thetracking is terminated (step S162) in response to the manipulation onthe tracking button 1432 again by the operator.

Therefore, the operator can obtain required information fast andsecurely by performing simple manipulations while he/she is performingsurgery.

As a result, according to this embodiment, an easy-to-use objectobservation system can be obtained which can display a virtual imageintended by an operator as a reference image. Thus, the security of anoperation can be improved, which can largely contribute to the reductionof an operation time.

The object observation system may have a construction as shown in FIG.111. FIG. 111 is an entire configuration diagram of an objectobservation system according to a variation example of the thirteenthembodiment.

The object observation system 1401B has a system controller 1410Bintegrated to the virtual image creating section 1407 as shown in FIG.111.

The system controller 1410B includes a CT image DB section 1418 b, acommunication I/F 1413 b, a memory 1414 b, a CPU 1415 b and a displayI/F 1416 b. The CT image DB section 1418 b performs the same operationsas those of the CT image DB section 1418 of the virtual image creatingsection 1407. The communication I/F 1413 b is connected to the lightsource apparatus 1403, the CCU 1405, the medical equipment 1409, the VTR1412, the inclination angle sensor 1411 and the remote controller 1417and also functions as the communication I/F 1421 of the virtual imagecreating section 1407. The memory 1414 b also functions as the memory1419 of the virtual image creating section 1407. The CPU 1415 b isconnected to the mouse 1423, the keyboard 1424 and the remote controller1417 and also functions as the CPU 1420 of the virtual image creatingsection 1407. The display I/F 1416 b is connected to the endoscopemonitor 1406 and the VR monitor 1408 and also functions as the displayI/F 1422 of the virtual image creating section 1407.

Since the object observation system 1401B has substantially the sameconstruction and operations as those of the thirteenth embodiment exceptthat the system controller 1410B also functions as the virtual imagecreating section 1407, the description thereof will be omitted herein.

Thus, since the object observation system 1401B can obtain substantiallythe same advantages as those of the thirteenth embodiment, and since thesystem controller 1410B can also function as the virtual image creatingsection 1407, the object observation system 1401B can be reduced in sizeas a whole and can be constructed at low costs.

Fourteenth Embodiment

FIGS. 112 and 113 relate to a fourteenth embodiment of the invention.FIG. 111 is an entire configuration diagram showing an objectobservation system according to the fourteenth embodiment. FIG. 112 is aflowchart showing processing operation, which is a feature of thefourteenth embodiment.

While the thirteenth embodiment has the remote controller 1417 to bemanipulated and used to instruct by an operator as virtual image changeinstructing means, the fourteenth embodiment has a microphone to bemanipulated and used to instruct by an operator as the virtual imagechange instructing means. Since the other components are the same asthose of the thirteenth embodiment, the description thereof will beomitted. The same reference numerals are given to the same components inthe description.

In other words, as shown in FIG. 112, an object observation system 1401Caccording to the fourteenth embodiment includes a system controller1410C connecting to the microphone 1451 for capturing voice of anoperator. As described later, the microphone 1451 can be used forinputting inserting point information and focus-point information of theendoscope 1402 with respect to a body to be examined as observationinformation input means.

The microphone 1451 is, for example, mounted on the head set, not shown,to be attached to the head of an operator and is removably connected tothe system controller 1410C. The microphone 1451 may be a pinmicrophone, which can be attached to an operator.

The system controller 1410C has a microphone I/F 1452 connecting to themicrophone 1451 and a voice recognizing portion 1453 forsignal-converting voice signals received by the microphone I/F 1452,recognizing the voice command and outputting a command signal inaccordance with the recognized voice command to the CPU 1415 c.

The rest of the construction is substantially the same as that of thethirteenth embodiment, and the description thereof will be omittedherein.

Then, in the object observation system 1401C, the CPU 1415 c of thesystem controller 1410C controls the entire system under the voicecontrol of an operator through the microphone 1451.

In the same manner as that of the thirteenth embodiment, the objectobservation system 1401C can perform image processing and displayprocessing on virtual image data and image change processing on virtualimages such as zoom-in, zoom-out and organ removal in response to inputsof an inserting point and a focus point under the voice control of anoperator during surgery through the microphone 1451.

A processing operation, which is a feature of the fourteenth embodiment,is shown in FIG. 113.

In the flowchart shown in FIG. 113, the object observation system 1401Cis powered on in order to perform surgery on a body to be examinedwithin the abdomen area of a patient so that voice input to the systemcontroller 1410C through the microphone 1451 can be performed, which isthe start of a voice input (step S170). Then, the operatorhimself/herself inputs an inserting point and a focus point by voice(steps S171 and S172), which is an observation information inputprocess.

Like the thirteenth embodiment, inputting an inserting point and a focuspoint (steps S171 and S172) may be performed by a nurse or an operatorby using the mouse 1423 or the keyboard 1424.

The subsequent operations (steps S173 to S172) are the same as thoseaccording to the thirteenth embodiment except that other commands arevoice-input by an operator himself/herself.

As a result, in addition to the same advantages as those of thethirteenth embodiment, the object observation system 1401C according tothe fourteenth embodiment can be easily controlled by voice without theinconvenience of remote control manipulations and can have goodoperability and a simple construction at low costs.

Fifteenth Embodiment

FIGS. 114 to 122 relate to a fifteenth embodiment of the invention. FIG.114 is an entire configuration diagram showing an object observationsystem of the fifteenth embodiment. FIG. 115 is a construction diagramshowing a construction of an operator's remote controller in FIG. 114.FIG. 116 is a screen display example of a virtual image display screenin a three-dimensional display form, which is displayed on a VR monitorin FIG. 114. FIG. 117 is a screen display example on which a virtualimage is displayed in a virtual image display area in FIG. 116. FIG. 118is a screen display example of a virtual image display screen in atwo-dimensional display form, which is displayed on the VR monitor inFIG. 114. FIG. 119 is a screen display example of an equipment settinginformation screen displayed on the VR monitor in FIG. 114. FIG. 120 isa flowchart showing a processing operation, which is a feature of thefifteenth embodiment. FIG. 121 is a screen display example of a virtualimage display screen for illustrating an operation of this embodiment.FIG. 122 is a screen display example of a virtual image display screenon which the virtual image in FIG. 121 is enlarged.

According to the thirteenth and fourteenth embodiments, a virtual imagecorresponding to an endoscopic live image can be displayed based oninclination angle data detected by an inclination angle sensor bytracking during surgery with the inclination angle sensor 1411 in theendoscope 1402. On the other hand, according to the fifteenthembodiment, an operator can freely input an inserting point and a focuspoint without tracking by using a remote controller for inputting aninserting point and a focus point as observation information inputmeans. The rest of the construction is the same as that of thethirteenth embodiment, and the same reference numerals are given to thesame components for description.

In other words, as shown in FIG. 114, an object observation system 1401Daccording to the third embodiment has an operator's remote controller1417D, a virtual image creating section 1407D and a system controller1410D. The operator's remote controller 1417D can be used for inputtingan inserting point and a focus point as observation information inputmeans. The virtual image creating section 1407D creates a virtual imageby performing image processing on virtual image data based on insertingpoint data and focus-point data input through the remote controller1417D. The system controller 1410D controls the virtual image creatingsection 1407D.

The system controller 1410D has a CPU 1415 d for controlling a CPU 1420d of the virtual image creating section 1407D based on an operationinstruction through the remote controller 1417D on a virtual imagedisplayed on the VR monitor 1408.

The CPU 1420 d of the virtual image creating section 1407D creates avirtual image by performing image processing on virtual image data basedon inserting point data and focus-point data input through the remotecontroller 1417D.

As shown in FIG. 115, for example, the remote controller 1417D includesan endoscope equipment operation portion 1460A, a 2D display operationportion 1460B, a 3D display operation portion 1460C and a settingoperation portion 1460D. The endoscope equipment operation portion 1460Achecks and sets an operation of an endoscope equipment. The 2D displayoperation portion 1460B is used for implementing two-dimensional display(2D display) of a virtual image on a display screen of the VR monitor1408. The 3D display operation portion 1460C is used for implementingthree-dimensional display (3D display) of a virtual image.

The endoscope equipment operation portion 1460A includes a white balancebutton 1461 a, a pneumoperitoneum button 1461 b, a pressure button 1461f, a record button 1461 c, a freeze button 1461 d and a release button1461 e. The white balance button 1461 a can be used for a display imagedisplayed on the endoscope monitor 1406. The pneumoperitoneum button1461 b can be used for executing a pneumoperitoneum apparatus 1409 a. Apressure button 1461 f can be used for increasing and decreasingpressure for establishing a pneumoperitoneum. The record button 1461 ccan be used for recording endoscopic live images in the VTR 1412. Thefreeze button 1461 d and the release button 1461 e can be used whenrecording is implemented.

The 2D display operation portion 1460B includes an axial button 1462 a,coronal button 1462 b and sagittal button 1462 c, which are compliantwith different kinds of 2D display modes.

The axial button 1462 a can be used for display an axial plane havingupper (head) and lower (foot) divisions of a body. The coronal button1462 b can be used for displaying a coronal plane having front (front)and rear (back) divisions of a body with respect to the major axis. Thesagittal button 1462 c can be used for displaying a sagittal planehaving left and right divisions of a body.

The 3D display operation portion 1460C includes an inserting pointbutton 1463 a, a focus button 1463 b and an image change operationportion 1431. The inserting point button 1463 a can be used forinputting an inserting point as a direction of a line of vision. Thefocus button 1463 b can be used for inputting a focus point. The imagechange operation portion 1431 is the same as the one according to thethirteenth embodiment.

The 3D display operation portion 1460C includes the same image changeoperation portion 1431 as the image change operation portion 1431 of theremote controller 1417 according to the thirteenth embodiment.

The setting operation portion 1460D includes an operation button 1464 aand a numeric keypad 1464 b. The operation button 1464 a and the numerickeypad 1464 b can be used for switching and/or determining setting inputinformation and for inputting numeric values and so on, respectively,for an operation setting mode determined by the endoscope apparatusoperation portion 1460A, the 2D display operation portion 1460B and the3D display operation portion 1460C.

An operator can use the remote controller 1417D including theseoperation portions 1460A to 1460D to obtain desired information fast.

With the object observation system 1401D according to this embodiment,the virtual image display screen 1440D is displayed on the displayscreen of the VR monitor 1408 as shown in FIG. 116, for example.

The virtual image display screen 1440D has the same construction as thatof the virtual image display screen 1440 according to the thirteenthembodiment except that a switched display portion 1465 on the upper partclose to the right end of the screen. The switched display portion 1465has a 2D mode display portion 1465 a for indicating 2D display ofvirtual images and a 3D mode display portion 1465 b for indicating 3Ddisplay of virtual images.

A direction of a line of vision of virtual images is determined inaccordance with an inserting point and focus point input by an operatorby manipulating (the numeric keypad 1464 b of) the remote controller1417D when the virtual image display screen 1440D shown in FIG. 116 isdisplayed on the display screen of the VR monitor 1408. Thus, virtualimages are displayed in 3D display form on the virtual image displayarea 1441 as shown in FIG. 117.

On the other hand, in order to check a state of a body to be examined ona 2D display, the virtual image display screen 1440D shown in FIG. 117can be switched to the virtual image display screen 1440E in 2D displayas shown in FIG. 118 in response to a manipulation on (the operationbutton 1464 a of) the remote controller 1417D by an operator. As aresult the virtual image display screen 1440E in 2D display form can bedisplayed on the display screen of the VR monitor 1408.

As shown in FIG. 118, the virtual image display screen 1440E includes a2D image display area 1441E on the center of the screen and an operationsetting area 1443E on the right end of the screen. The 2D image displayarea 1441E displays a virtual image two-dimensionally. The operationsetting area 1443E can be used for manipulating and/or setting the 2Dimage display area 1441E.

The operation setting area 1443E includes a switched display portion1465, which is the same as that of the virtual image display screen1440D, on the upper part. The lower part of the switched display portion1465 includes an axial display switch 1466 a, coronal display switch1466 b and sagittal display switch 1466 c, which are compliant with the2D display modes.

On the virtual image display screen 1440E, one of the display switches(that is, the axial display switch 1466 a, coronal display switch 1466 band sagittal display switch 1466 c) in the operation setting area 1443Ecan be selected in accordance with a manipulation on a respective 2Ddisplay mode button (of the axial button 1462 a, the coronal button 1462b and the sagittal button 1462 c) of the remote controller 1417D by anoperator. Thus, a virtual image in a selected 2D display mode isdisplayed two-dimensionally on the 2D image display area 1441E.

On the other hand, in order to check an operation and settings of theendoscope apparatus, the virtual image display screen 1440D shown inFIG. 117 can be switched to the equipment setting information screen1470 as shown in FIG. 119, which is displayed on the display screen ofthe VR monitor 1408, in response to a manipulation on (the operationbutton 1464 a of) the remote controller 1417D by an operator.

As shown in FIG. 119, the equipment setting information screen 1470includes, for example, a patient's name display portion 1471 on theupper part of the screen, a pneumoperitoneum display portion 1472,electric knife display portion 1473, ultrasonic treatment displayportion 1474, VTR display portion 1475, camera intensity adjustingportion 1476, CO2 capacity display portion 1477, CCU operation displaypotion 1478 and live image display portion 1479 under the patient's namedisplay portion 1471 and a setting input display portion 1480 on thelowest part of the screen. The patient's name display portion 1471 showsa patient's name. The pneumoperitoneum display portion 1472 showsinformation such as an operation state, a pneumoperitoneum pressure anda temperature of the pneumoperitoneum apparatus 1409 a. The electricknife display portion 1473 shows a setting and operation state of theelectric knife apparatus 1409 b. The ultrasonic processing displayportion 1474 shows an ultrasonic output state by the ultrasonictreatment apparatus 1409 c. The VTR display portion 1475 shows aremaining amount of a tape in the VTR 1412. The camera intensityadjusting portion 1476 shows an intensity adjustment (iris) state of thecamera head 1404. The CO2 capacity display portion 1477 shows a totaloutput capacity (the integral of the capacity) of CO2 into a bodycavity. The CCU operation display portion 1478 shows an operation state(freeze, release or zoom) of the CCU 1405. The live image displayportion 1479 displays endoscopic live images. The setting input displayportion 1480 can be used for inputting settings of each equipment.

The setting input display portion 1480 includes an input switch 1481 anda function key portion 1482. The input switch 1481 is used for inputtingdifferent settings. Different setting modes are registered with thefunction key portion 1482 in advance.

The function key portion 1482 includes Functions F1 to F4. A whitebalance switch for implementing white balance is registered withFunction F1. A system record switch for implementing system recording isregistered with Function F2. A camera intensity-up switch for increasingthe camera intensity is registered with Function F3. A cameraintensity-down switch for decreasing the camera intensity is registeredwith Function F4.

In the equipment setting information screen 1470, an operator canmanipulate the endoscope apparatus operation portion 1460A of the remotecontroller 1417D, select one of different equipment settings to bedisplayed, input a numeric value as required so that the equipmentsetting information can be changed and/or set.

In the object observation system 1401D, the CPU 1415 d of the systemcontroller 1410D controls the entire system according to an operationinstruction signal from an operator through the remote controller 1417D.

Processing operations, which are features of the fifteenth embodiment,are shown in the flowchart in FIG. 120.

Here, surgery is performed on a body to be examined within the abdomenarea of a patient by using the object observation system 1401D shown inFIG. 114. In this case, when the object observation system 1401D haspower applied thereto, a nurse or an operator starts a program based ona control method for the object observation system of the invention,which is stored in the CPU 1420 d, by using the mouse 1423 or thekeyboard 1424 first of all.

Then an operator inserts the endoscope 1402 into the abdomen area of thepatient. The object observation system 1401D causes an endoscopic liveimage obtained by the endoscope 1402 to be displayed on the displayscreen of the endoscope monitor 1406 under the display control of theCPU 1415 d of the system controller 1410D as a body to be examined imageobtaining operation.

Then, the operator manipulates the setting operation portion 1460D ofthe remote controller 1417D, selects and inputs the 2D display mode or3D display mode of the endoscope apparatus operation mode or virtualimage display mode as a mode select command and performs manipulationswith the selected display mode.

The CPU 1415 d of the system controller 1410D judges the presence of theinput of a mode selection command based on an operation instruction on(the setting operation portion 1460D of) the remote controller 1417D byan operator (step S191) and, if yes, judges whether the mode selectioncommand is a 2D display mode or 3D display mode of the endoscopeequipment operation mode or the virtual image display mode (step S192).Based on the judgement result, the CPU 1415 d switches to the displaymode.

Here, if the 3D display mode of the virtual image display mode isselected and input, the CPU 1415 d of the system controller 1410Didentifies the input of the selection of the 3D display mode of thevirtual image display mode and switches to and displays the virtualimage display screen 1440D in the 3D display form as shown in FIG. 116on the VR monitor 1408.

Then, on the virtual image display screen 1440D in the 3D display form,an operator needs to input an inserting point and a focus point bymanipulating the remote controller 1417D.

First of all, the operator selects and inputs thedirection-of-line-of-vision input command by manipulating the operationbutton 1464 a of the remote controller 1417D and inputs numeric valuesfor an inserting point and focus point by manipulating the numerickeypad 1464 b.

Thus, the CPU 1415 d of the system controller 1410D recognizes the inputof the selection of the direction-of-line-of-vision input command (stepS193), inputs an inserting point and a focus point based on the numericvalues input from the numeric keypad 1464 b (step S194), and determinesthe direction of the line of vision (step S195). In other words, thestep S194 is an observation information input process.

Then, the CPU 1415 d of the system controller 1410D controls the CPU1420 d of the virtual image creating section 1407D to perform imageprocessing on virtual image data in accordance with the determineddirection of the line of vision and to display the virtual image in thevirtual image display area 1441 as shown in FIG. 117 (step S196). Inother words, the step S196 is a virtual image processing process.

After that, the operator needs to perform image change processing on thevirtual image displayed in the virtual image display area 1441.

Here, for example, the virtual display screen 1440D is displayed on thedisplay screen of the VR monitor 1408 as shown in FIG. 121. Then, inorder to increase the display scale of the virtual image, the operatormanipulates the zoom-in button 1431 b of the remote controller 1417D asan image processing change command.

Thus, the CPU 1415 d of the system controller 1410D recognizes theselection of the image processing change (step S193) and controls theCPU 1420 d of the virtual image creating section 1407D to zoom-in thevirtual image currently displayed in the virtual image display area 1441in response to the manipulation on the zoom-in button 1431 b of theremote controller 1417D as the image change processing (step S197)corresponding to the input command and to display the virtual image inthe virtual image display area 1441 as shown in FIG. 122. In otherwords, the step S197 is a virtual image change process.

On the other hand, in order to check the state of the body to beexamined in 2D display mode, the operator selects and inputs the 2Ddisplay mode by manipulating the operation button 1464 a of the remotecontroller 1417D.

Thus, the CPU 1415 d of the system controller 1410D recognizes the inputof the selection of the 2D display mode (step S193), switches to anddisplays the virtual image display screen 1440E in the 2D display modeshown in FIG. 118 on the display screen of the VR monitor 1408.

Then, with reference to the virtual image display screen 1440E, theoperator manipulates the operation buttons of the 2D display operationportion 1460B of the remote controller 1417D.

Thus, the CPU 1415 d of the system controller 1410D controls the CPU1420 d of the virtual image creating section 1407D to display thevirtual image in the 2D display mode corresponding to the input command(step S198).

On the other hand, in order to check and set an operation of theendoscope apparatus during surgery, the operator manipulates theoperation button 1464 a of the remote controller 1417D and selects andinputs the endoscope operation mode.

Thus, the CPU 1415 d of the system controller 1410D recognizes the inputof the selection of the endoscope apparatus operation mode (step S193)and switches to and displays the equipment setting information screen1470 shown in FIG. 199 on the display screen of the VR monitor 1408.

Then, the operator changes and/or sets equipment setting information bymanipulating buttons for the endoscope apparatus 1460A on the remotecontroller 1417D with reference to the equipment setting informationscreen 1470.

Thus, the CPU 1415 d of the system controller 1410D operates theendoscope apparatus in accordance with the input command (step S199).

After that, the processes from the step S191 are repeated to the end ofthe operation (step S200).

The commands may be input by a nurse or an operator by using the mouse1423 or the keyboard 1424.

As a result, with the object observation system 1401D according to thefifteenth embodiment having the remote controller 1417D allowing theinput of an inserting point and a focus point, an operator can freelyinput an inserting point and a focus point and view a virtual image of adesired area in addition to the same advantages as those of thethirteenth embodiment, which can advantageously improve the operability.

Furthermore, with the object observation system 1401D according to thefifteenth embodiment, an operator can freely check and set an operationof the endoscope apparatus by using the remote controller 1417D, whichcan advantageously further improve the operability.

Sixteenth Embodiment

FIGS. 123 and 124 relate to a sixteenth embodiment of the invention.FIG. 123 is an entire configuration diagram showing an objectobservation system according to the sixteenth embodiment. FIG. 124 is aflowchart showing a processing operation, which is a feature of thesixteenth embodiment.

While the remote controller 1417D to be manipulated by an operator isprovided as virtual image change instructing means according to thefifteenth embodiment, a microphone to be used by an operator forinstructing a manipulation is provided as virtual image changeinstructing means according to the sixteenth embodiment. Since the othercomponents are the same as those of the fifteenth embodiment, thedescription thereof will be omitted. The same reference numerals aregiven to the same components for description.

In other words, an object observation system 1401E according to thesixteenth embodiment includes a system controller 1410E to which themicrophone 1451E is connected as shown in FIG. 123.

For example, the microphone 1451E is attached to a head set (not shown)to be attached to the head of an operator and is removably connected tothe system controller 1410E. The microphone 1451E may be a pinmicrophone, which can be attached to an operator.

The system controller 1410E has a microphone I/F 1452 e connecting tothe microphone 1451E and a voice recognizing portion 1453 e forsignal-converting voice signals received by the microphone I/F 1452 e,recognizing the voice command and outputting a command signal inaccordance with the recognized voice command to the CPU 1415 e.

The rest of the construction is substantially the same as that of thefifteenth embodiment, and the description thereof will be omittedherein.

Then, in the object observation system 1401E, the CPU 1415 e of thesystem controller 1410E controls the entire system under the voicecontrol of an operator through the microphone 1451E.

In the same manner as that of the fifteenth embodiment, the objectobservation system 1401E can be operated in a display mode selectedbetween the 2D display mode and the 3D display mode of one of theendoscope apparatus mode and the virtual image display mode under thevoice control of an operator during surgery through the microphone1451E.

A processing operation, which is a feature of the sixteenth embodiment,is shown in FIG. 124.

In the flowchart shown in FIG. 124, the object observation system 1401Eis powered on in order to perform surgery on a body to be examinedwithin the abdomen area of a patient so that voice input to the systemcontroller 1410E through the microphone 1451E can be performed, which isthe start of a voice input (step S300). Then, the operatorhimself/herself inputs each command by voice.

The subsequent operations (steps S191 to S200) are the same as thoseaccording to the fifteenth embodiment except that other commands arevoice-input by an operator himself/herself.

Like the fifteenth embodiment, inputting each command may be performedby a nurse or an operator by using the mouse 1423 or the keyboard 1424.

As a result, in addition to the same advantages as those of thefifteenth embodiment, the object observation system 1401E according tothe sixteenth embodiment can be easily controlled by voice without theinconvenience of remote control manipulations and can have goodoperability and a simple construction at low costs.

As described above, according to this embodiment, an easy-to-use objectobservation system and method of controlling an object observationsystem can be obtained which can display a virtual image intended by anoperator as a reference image.

The invention is not limited to the first to sixteenth embodiments, andvarious changes and modifications of the invention can be made withoutdeparting from the spirit and scope of the invention.

1-14. (canceled)
 15. An object observation system, comprising: anendoscope having an insert portion, which can be used for observing abody to be examined; at least one treating device for performing atreatment on the body to be examined; a first detecting portion fordetecting information indicating an observation direction of the insertportion of the endoscope; a second detecting portion for detectinginformation indicating a treatment direction of the treating device; athree-dimensional image data storing portion for storingthree-dimensional image data relating to the body to be examined; athree-dimensional image data processing portion for creating first andsecond three-dimensional image data corresponding to the first andsecond detecting portions by processing the three-dimensional image databased on information detected by the first and second detectingportions; first and second three-dimensional image display portions,which can display first and second three-dimensional images based on thefirst and second three-dimensional image data and an endoscopicobservation image by the endoscope; a switching section, which canselectively switch and output, to the first and second three-dimensionalimage display portion, the first and second three-dimensional image datafrom the three-dimensional image data processing portion; and a controlportion for controlling the switching section.
 16. An object observationsystem according to claim 15, wherein the first and second detectingportions have operation portions for instructing display modes of thefirst and second three-dimensional image display portions. 17-40.(canceled)
 41. A control method of an object observation systemincluding an endoscope having an insert portion, which can be used forobserving a body to be examined and at least one treating device forperforming a treatment on the body to be examined, the methodcomprising: a first step of detecting information indicating anobservation direction of the insert portion of the endoscope; a secondstep of detecting information indicating a treatment direction of thetreating device; a step of storing three-dimensional image data relatingto the body to be examined; a step of creating first and secondthree-dimensional image data corresponding to the first and seconddetecting steps by processing the three-dimensional image data based oninformation detected by the first and second detecting steps; a step ofdisplaying first and second three-dimensional images based on the firstand second three-dimensional image data and an endoscopic observationimage by the endoscope; a step of selectively switching the first andsecond three-dimensional image data for the display step; and a step ofcontrolling the switching step.
 42. A control method of an objectobservation system according to claim 41, wherein the first and secondsteps have operation steps for instructing display modes of the displaysteps. 43-52. (canceled)